VSS providers conflicts.
In case you're using Symantec BackupExec 2010 as a backup solution for your Virtual Infrastructure running Vmware vSphere 4, you might be founding yourself in curious situations when your backup jobs might finish right (but sometimes not).
By digging deep inside of log files together with Symantec support, we found an issue when VSS providers of BE 2010 conflicts with VSS providers of Windows VM. With that conflict we had, the manual snapshots of VMs don't work anymore issuing “protocol errors in VMX”.
Note that in this particular case the VMware's Volume shadow copy component present inside of the default installation of VMware Tools was NOT installed and as for the quiescing mechanism the Symantec's VSS is used, BUT it must be shown by issuing the “vssadmin list providers” command only DURING the backup process.
So the example below is correct – only one VSS provider is listed. That was not our case…..we had 2 of them. And no backups were running…
To readjust the situation you must follow a KB from Symantec providing a script located under the installation folder of Symantec's agent situated inside of the VM. This script unregisters the Symantec's VSS provider.
To resolve this, perform the following:
01) Run the “post-thaw-script.bat” from the VSS Provider directory in the Backup Exec install directory of the guest machine (Default path is C:\Program Files\Symantec\Backup Exec\RAWS\VSS Provider)
02) Reboot the guest machine
https://seer.entsupport.symantec.com/docs/352025.htm