Recently we have gone through the process of upgrading our Windows 6.0 vCenter Server with external SQL to vCSA 6.5. I must say now how good the entire process was from start to finish, VMware have really done themselves proud on that tool. Our environment isn’t huge but it is big enough that we thought we might see problems – but no!

Part of the migration work was to get backups up and runnign as they were with our Windows vCenter (if not slightly different/better). My understanding is that the supported method for backup is to use the VAMI interface and run a full “file dump” backup of the vCSA with which you can restore into any blank deployed vCSA and you are back in the game. We have a Rubrik for snapshotting but using the VMware method is of course supported and preferred.

The Issue

Upon using the VMware provided Bash Script we encountered the following error in the backup.log file that is produced:

“{“type”:”com.vmware.vapi.std.errors.unauthenticated”,”value”:{“messages”:[{“args”:[],”default_message”:”Unable to authenticate user”,”id”:”vapi.security.authentication.invalid”}]}}”

Further investigation showed further errors in the VAPI endpoint log

We could run a manual backup from the VAMI interface as the root user but just not using the bash script which is essentially using the VAMI API to curl a request to run a backup. The error above seems related to “authentication_sso.py” and being unable to validate the signing chain signature. Without further help there was no way I was going in to modify or look at that script on my own on a now Production vCSA.

I also created a seperate master user in the @vsphere.local domain to test running the backups but still had no luck.

I ran the script manually and the problem occured at the start of the POST to the appliances rest API.

The Fix

After speaking with several smart people in the vExpert slack channel, I raised a case with VMware support. I eventually received a response which told me to edit the following file:

There is a value that needed changing from:

To the following:

Be careful with the amendment, there is space indentation on the code and there must be exactly 8 spaces in from the new line

Then a simple stop and start of the applmgmt service to apply the fix:

Now the script runs perfectly daily to our backup respository. I believe this might become defunct in vSphere 6.7 as I think there is now a GUI way of scheduling backups!