VMGuru

For creating a backup of a VMware VM with an iSCSI connected volume you can use a Veeam Agent for Windows (VAW). A specific case at a customer last week sparked my interest because they pinged me why they kept failing to do a proper backup of a VMware VM with a iSCSI volume connected to it from a QNAP.

Troubleshooting

One symptom that was visible is that the backup runs for an extremely long time and that the proxy process was struggling in retrieving and processing the data on that particular volume.

First question to ask was: Are you backing up the VM with a Veeam Agent installed in the Operating System or try to do a backup through the hypervisor, so agentless?

He installed a Veeam agent properly, but the backup kept complaining about that the disk connected was running low on free space.

Subsequent error message around the disk was: Error Volume Data (G:) does not have enough free disk space to create VSS snapshot. We checked how large the mounted volume was, because volumes larger than 64TB are not supported by Microsoft VSS. The volume is 5TB with 1.5TB in use.

Next check was if the volume perhaps was a thin provisioned volume, which was not the case. So, there should be enough free space to complete the request to create a VSS snapshot on that volume you would think.

Then it might be a security issue with the volume connected. I asked the customer to raise a case with Veeam support so they could run through the Veeam logfiles to get some more insight in what is happening in the background process.

Logs

The log file held several messages as shown below:

[09.10.2019 09:30:06] <01> Info [EpFreeSpaceHelper] Volume [‘Archief (G:)’, isSystem=False] length is 5497421824000 bytes. Free space required: 1073741824 bytes
[09.10.2019 09:30:06] <01> Error Volume Data (G:) does not have enough free disk space to create VSS snapshot
[09.10.2019 09:30:07] Info <1d7c> The following volumes should be added to snapshot:
[09.10.2019 09:30:07] Info <1d7c> Volume ‘{32df8165-3311-43bb-bcc4-fb0ff0965e9d}’, mount point ‘G:\’.
[09.10.2019 09:30:07] Info <1d7c> Enumerating volumes and disks for desktop backup.
[09.10.2019 09:30:07] Info <1d7c> Enumerating non shadow fixed volumes.
[09.10.2019 09:30:07] Error <1d7c> Unable to retrieve free space size from file system on volume ‘\\?\Volume{32df8165-3311-43bb-bcc4-fb0ff0965e9d}\’.
[09.10.2019 09:30:07] Warning <1d7c> Win32 error: Access is denied. Code: 5

At the same time of the entries in the log file a lot of events are written to the Windows Application log with event id: 12290.

Log Name: Application
Source: VSS
Date: 9-Oct-19 9:30:06 AM
Event ID: 12290
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: FS001.Veeam.Local
Description:
Volume Shadow Copy Service warning: GetDiskFreeSpaceEx() for \\?\Volume{32df8165-3311-43bb-bcc4-fb0ff0965e9d}\ failed with 0x00000005. hr = 0x00000000, The operation completed successfully.
Please ensure that you have enough free space on Volume {32df8165-3311-43bb-bcc4-fb0ff0965e9d} for creating the snapshot.

Error code 0x00000005 in the description is an access denied error.
The last line in the log example: Win32 error: Access is denied. Code:5 combined with the event id 12290 and error code 0x00000005 gave me the confirmation I needed to follow-up with running through the security settings of the volume.

Conclusion

While checking the Security settings on the volume we noticed that SYSTEM was removed from the list that had access to the volume. Somebody was a bit too rigorous in hardening the volume!

After adding SYSTEM again with Full access control the backup returned to green and finished in minutes instead of taking hours or failing all together.