I’ve been attempting to test out the SCP option in the new File-based VCSA backup feature in 6.5 (specifically, I’m testing against 6.5.0e which is the latest as of today). My backup target is currently a Windows box running Bitvise SSH Server using a dedicated local account that is limited to file transfers only (no shell) and a virtual root directory. The user config looks like this if anyone wants to try and reproduce locally.
I have verified that SCP file transfers to and from my VCSA to this SSH server using the account I configured work just fine. That is, I can SSH to vCenter, drop to the shell, and “scp filename backup@myserver:/” or “scp backup@myserver:/filename ./” and everything transfers successfully.
However, when I attempt to configure the backup job from the VCSA web UI, I get the following error.
Access to the remote server is denied. Check your credentials and permissions.
For reference, here are the values I used in the wizard. I also pre-created an empty “myfolder” in the virtual root.
When I check the logs on the SSH server, I see that the session was denied because it was attempting to execute a command, “Command execute request rejected.” This is exactly what I wanted it to do when I set the “No shell access” option. But why is VCSA trying to open a shell when it’s supposed to be using SCP?
So let’s dig into the log on the VCSA side. I see this in /var/log/vmware/applmgmt/backupRestoreAPI.log:
[MainProcess:PID-2311] ERROR: Failed to validate SSH login at 10.10.10.10:22/myfolder; sshCmd:[‘sshpass’, ‘-p’, ‘****’, ‘ssh’, ‘-o’, ‘ConnectTimeout=10’, ‘-o’, ‘StrictHostKeyChecking=no’, ‘-p’, ’22’, ‘-q’, ‘-l’, ‘****’, ‘10.10.10.10’, ‘exit’], rc: 255, stdOut: , stdErr:
That definitely verifies that it was trying to open a shell. But why? Just to validate credentials in the wizard? Why not just SCP a zero byte file or something? Again, I’ve already verified that plain old scp works just fine with the current configuration and the GUI and Docs all refer to this option specifically as “SCP” and not “SSH”.
In any case, now I’m curious what it will take to get it to work. So I modify the settings for the backup user to allow a shell. Bitvise has a few options for shells. But the only one that respects the virtual filesystem root folder is their BvShell option. From their help docs:
BvShell is a bash-like shell provided by the SSH Server. This is the only shell type which respects the virtual filesystem layout configured for the client in SSH Server settings. BvShell does not permit execution of arbitrary programs. It is intended to be enabled for clients limited to file transfer access that may wish to use commands (e.g. file copy) that their SFTP or SCP client might not provide. (emphasis mine)
Sounds great! The backup user should still be tied to the virtual root folder and limited to commands that are associated with file transfers. I run through the wizard again and make it through to where it seems to be starting the backup job. But it eventually errors out with the message,
BackupManager encountered an exception. Please check logs for details.
So, back to the logs. backupRestoreAPI.log is no help. It just confirms the job failed. There’s now a /var/log/vmware/applmgmt/backup.log though which has more details.
[MainProcess:PID-60326] ERROR: Failed to create dir /myfolder/ at 10.10.10.10; sshCmd:[‘sshpass’, ‘-p’, ‘****’, ‘ssh’, ‘-o’, ‘ConnectTimeout=10’, ‘-o’, ‘StrictHostKeyChecking=no’, ‘-p’, ’22’, ‘-q’, ‘-l’, u’****’, u’10.10.10.10′, u’mkdir -p /myfolder/’], rc:2, stdOut:, stdErr:mkdir: Unexpected command parameter: -p.
So it’s trying to create the “myfolder” which I had already pre-created using “mkdir -p” and apparently the mkdir command in BvShell doesn’t support the “-p” option. So now we know not only does this “SCP” transfer method require a shell to execute commands. It is also expecting to have access to (presumably) GNU compatible binaries.
None of our other shell choices in Bitvise respect the virtual filesystem root. So at this point, we’re not going to satisfy the security folks even if we do manage to make this work. But in the interest of science, let’s see if we can go further and make it work. The “Command Prompt” and “Powershell” options are going to be no good without installing Windows ports of those utilities or resorting to Cygwin. “Git access only” won’t work either. But we do have a “bash” option that just needs to be pointed to an executable and this server already has Git for Windows installed which comes with Git Bash that might just do the trick. At the very least, it has a GNU mkdir that supports “-p”.
So we change the backup user’s shell to “bash” and point it to “C:Program FilesGitbinbash.exe”. We also change the virtual filesystem layout to “Allow full access” and update the Location we’re using in the Backup GUI to “10.10.10.10/c/temp/backup/myfolder”. We make it through the Backup wizard GUI again and the job starts…and then fails with,
Error at process ComponentScriptsBackup; rc:1. For details please check the log file: /var/log/vmware/applmgmt/backup.log.
*sigh* Back to the log file where we find a lot more going on but there’s some sort of error with cat following a tar command?
[LotusBackup:PID-6322] INFO: BackupLotus: Dispatching files [‘lock.mdb’, ‘data.mdb’]
[LotusBackup:PID-6322] INFO: tarCmd = tar -cz -C /tmp/backup_lotus/ –ignore-failed-read –warning=”no-file-ignored” “lock.mdb” “data.mdb”
[ComponentScriptsBackup:PID-6324] ERROR: rc: 1, stderr: cat: ‘>’: No such file or directory
cat: /c/temp/backup/myfolder/imagebuilder.gz.enc: No such file or directory
[ComponentScriptsBackup:PID-6324] ERROR: Component imagebuilder backup dispatch failed.
At this point I’m spent and I keep coming back to the same question. Why is this backup option labeled “SCP” if it’s not actually going to use SCP??? There’s nothing in the documentation that references any requirements for a GNU compatible shell or even a shell at all. The only notes about SCP at all are reminding you to use the absolute path to the folder. Why should SCP work any differently than FTP where it is clearly able to just transfer the backup files???
