While we had some hardware errors and failures on our physical Veeam proxie and repository servers in the past, we tought it was a good idea to get them at least backud upped regularly through Veeam’s own Physical & Cloud Infrastructure. Yes, we did put them in a protection group. How insignificant it may seem, it give us some advantages like recovery media (ISO), different restore possibilities etc. Just in case we might need it in the future. It must be said that we are only backuing up the operating system and the C: and D: volumes, not the backup data itself.
So the idea was to backup one proxy and repository from the A. datacenter to the repository server in the B. datacenter and vice versa. So, after configuring the protection group, installing the driver and creating the backup job, one repository server backup job failed and kept on failing with the following message:
7/23/2019 12:51:55 PM :: Creating VSS snapshot Error: Failed to create snapshot: Backup job failed.
Cannot create a shadow copy of the volumes containing writer's data.
VSS asynchronous operation is not completed. Operation: [Shadow copies commit]. Code: [0x80042306].
This could only mean that on first hand we have a problem with the VSS (Volume Shadow Copy service) or something related to thatm and that could mean a lot of other things. So first I started to look at the VSS writers on the repository server with vssadmin list writers :
As you can see in the above screen capture, the last error is : Timed Out. So lets take a look at the event viewer.
So it has something to do with VSS; accounts and permissions perhaps? So let’s fire up DCOMCNFG and lets find out if there is a problem with an account or a permission.
Everything looks allright and even the registry setting of the VSSAccessControl seems to be OK:
By now it is time to take a look at some Veeam logging to see what is actualy going on, and to give us some more indepth knowledge of the problem.
So the log file to look for would be the name of the job in which we are backing up the physical machine, in this case the name of the log file is Job.PRD_Physical_Backup_Prot-Grp-Name_-_extension.log.
Here is what we have found:
The process of committing shadow copies is failing.
[22.07.2019 15:22:24] Info < a60> Commiting shadow copies.
[22.07.2019 15:22:25] Info < a60> Commiting shadow copies.. Failed.
[22.07.2019 15:22:25] Info < a60> Cancelling changes made to prepare an application for a subsequent restore operation. Application: [NTDS].
[22.07.2019 15:22:25] Info < a60> There is no changes related to the NDIS service or changes were already cancelled.
[22.07.2019 15:22:25] Info < a60> Cancelling changes made to prepare an application for a subsequent restore operation. Application: [NTDS].. Ok.
[22.07.2019 15:22:25] Info < a60> Cancelling changes (if exists) that were made before the freeze stage in order to support Exchange server writer.
[22.07.2019 15:22:25] Info < a60> Cancellation of changes made to support Exchange Server writer is not required.
[22.07.2019 15:22:25] Info < a60> Cancelling changes (if exists) that were made before the freeze stage in order to support Exchange server writer.. Ok.
[22.07.2019 15:22:25] Info < a60> Creating a shadow copy of the volumes containing writer's data. Backup type: 'VSS_BT_FULL'.. Failed.
[22.07.2019 15:22:25] Info < a60> Performing backup steps. Backup type: [VSS_BT_FULL].. Failed.
[22.07.2019 15:22:25] Info < a60> Terminating the snapshot set. Snapshot set ID: [{198b61f6-f167-473b-af85-5fda94efeddb}].
[22.07.2019 15:22:25] Info < a60> Terminating the snapshot set. Snapshot set ID: [{198b61f6-f167-473b-af85-5fda94efeddb}].. Ok.
[22.07.2019 15:22:25] Error < a60> Freeze job failed.
[22.07.2019 15:22:25] Warning < a60> Backup job failed.
[22.07.2019 15:22:25] Warning < a60> Cannot create a shadow copy of the volumes containing writer's data.
[22.07.2019 15:22:25] Warning < a60> VSS asynchronous operation is not completed. Operation: [Shadow copies commit]. Code: [0x80042306].
[22.07.2019 15:22:27] <01> Info Failed to create snapshot. Error code -2147467259. 'Backup job failed.
[22.07.2019 15:22:27] <01> Info Cannot create a shadow copy of the volumes containing writer's data.
Usually this might happen due to insufficient space for shadow copy storage.
As you now can see in the output of the vssadmin list shadowstorage command, listed is only the shadow copy storage for C: volume.
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.
Shadow Copy Storage association
For volume: (C:)?Volume{6f0bd9f1-4835-11e6-80b5-806e6f6e6963}
Shadow Copy Storage volume: (C:)?Volume{6f0bd9f1-4835-11e6-80b5-806e6f6e6963}
Used Shadow Copy Storage space: 0 bytes (0%)
Allocated Shadow Copy Storage space: 0 bytes (0%)
Maximum Shadow Copy Storage space: 36,5 GB (25%)
So what we need to do is to add to the snapshot set [?Volume{6f0bd9f0-4835-11e6-80b5-806e6f6e6963}] and [?Volume{6f0bd9f1-4835-11e6-80b5-806e6f6e6963}].
[22.07.2019 15:22:23] Info < a60> The following volumes should be added to the snapshot set.
[22.07.2019 15:22:23] Info < a60> {
[22.07.2019 15:22:23] Info < a60> [?Volume{6f0bd9f0-4835-11e6-80b5-806e6f6e6963}]. Mount point: []
[22.07.2019 15:22:23] Info < a60> [?Volume{6f0bd9f1-4835-11e6-80b5-806e6f6e6963}]. Mount point: [C:]
[22.07.2019 15:22:23] Info < a60> }
To add the shadow copy storage for the system reserved volume to the C: volume we need to use the below command (command prompt with administrative privileges):
vssadmin add shadowstorage /for=?Volume{6f0bd9f0-4835-11e6-80b5-806e6f6e6963} /on=C: /maxsize=unbounded
From there on, just restarted the job and everything went just fine…
Arie-Jan Bodde
