VMware Cloud Foundation (VCF): SFTP

Overview

If you are reading this, I bet you are trying to get your SDDC Manager configured to do backups via SFTP…and it’s just not working. You keep seeing a similar message in SDDC Manager, even though you are using all the correct configuration information.

Invalid parameter: Validation failed for directory path <configured backup directory> on server <Host FQDN or IP>. Make sure backup directory is intact and sftp server has write permissions on backup path .

I feel your frustration; this took way more effort than necessary to figure out and get configured. Also, why would the sftp server need write permissions on the backup path? (hint: it doesn’t — this is just a poorly written message) If I had to guess, I would think it is supposed to refer to the sftp user, not the server.

For reference, I am testing with Fedora Linux 39 (Server Edition) running OpenSSH 9.3p1
(openssh-9.3p1-10.fc39.src.rpm)

The following note is in the OpenSSH release notes for version 8.7:

OpenSSH will disable the ssh-rsa signature scheme by default in the next release. In the SSH protocol, the “ssh-rsa” signature scheme uses the SHA-1 hash algorithm in conjunction with the RSA public key algorithm.

Guess what still uses ssh-rsa? If you guessed SDDC Manager, you would be correct! This product, at the time of this writing, is at version 5.1.1, released on March 26, 2024. Can you guess when ssh-rsa was deprecated? In OpenSSH, version 8.8, released on September 26, 2021. (for interested parties, that’s 912 days or 2 years and six months!) It’s uncanny to me that this is even an issue with preventable cybersecurity configurations as simple as this one.

One thing I should mention, here. I implemented the fix on my SFTP Server. This means I lowered the security posture for the whole server, not just the offending device. I was not able to narrow the scope to just the offending device, unfortunately. So that leaves three options that I can think of.

1. Wait for Broadcom to update the way this is handled in the future and not have backups — this sucks.
2. Build a separate and specific SFTP server for just this one use case — also sucks.
3. Lower the security on the already available SFTP server — I don’t know which is the stronger case for this option, having backups of critical infrastructure or lowering the security posture on one server to administer.

In the release notes for OpenSSH 8.8, there is a method to apply to the user logging into the SFTP server. However, the SDDC Manager Backup Configuration is an automated process and this configuration did not work for me. I tried to create a local user and group on the SDDC Manager and add a local config file to the .ssh directory. Unfortunately, the log still showed errors and the configuration never saved.

Incompatibility is more likely when connecting to older SSH implementations that have not been upgraded or have not closely tracked improvements in the SSH protocol. For these cases, it may be necessary to selectively re-enable RSA/SHA1 to allow connection and/or user authentication via the HostkeyAlgorithms and PubkeyAcceptedAlgorithms options. For example, the following stanza in ~/.ssh/config will enable RSA/SHA1 for host and user authentication for a single destination host: Host old-host HostkeyAlgorithms +ssh-rsa PubkeyAcceptedAlgorithms +ssh-rsa

SFTP Server Configuration

Okay, enough yapping, let’s get to it.

My SFTP server is configured with the following at the bottom of the /etc/ssh/
sshd_config file. (Ref: https://italchemy.wordpress.com/2021/12/06/installing-sftp-server-oncentos-red-hat-fedora-ec2-linux-the-complete-guide/)

### ADDED BY ADMIN ###

Match User sftpuser
ForceCommand internal-sftp
PasswordAuthentication yes
ChrootDirectory /var/sftp
PermitTunnel no
AllowAgentForwarding no
AllowTcpForwarding no
X11Forwarding no

This is what makes SFTP possible for me in the first place. I am able to use SFTP software to successfully connect, so I know the configuration works.

The /etc/ssh/sshd_config contains the following block:

# To modify the system-wide sshd configuration, create a  *.conf  file under
#  /etc/ssh/sshd_config.d/  which will be automatically included below
Include /etc/ssh/sshd_config.d/*.conf

So I decided to take a peek at the directory to see what was in there. I found the following
configuration file:
/etc/ssh/sshd_config.d/40-redhat-crypto-policies.conf

When I opened the file, I realized this file actually adds another Include. But, look at the message and notice what it contains. I am going to emphasize the interesting line.

# This system is following system-wide crypto policy. The changes to
# crypto properties (Ciphers, MACs, ...) will not have any effect in
# this or following included files. To override some configuration option,
# write it before this block or include it before this file.
# Please, see manual pages for update-crypto-policies(8) and sshd_config(5).
Include /etc/crypto-policies/back-ends/opensshserver.config

On my Fedora system, this file started with 40, so I created a new file called, 30-sddc-manager-sftp.conf.

The contents of the file are simple. Notice I use the keyword, a space, and then an addition sign to append the ssh-rsa algorithm.

HostKeyAlgorithms +ssh-rsa
PubkeyAcceptedAlgorithms +ssh-rsa

Save the file and test the sshd configuration with sudo sshd -t. If there are no errors, the command line will just display a new prompt. If there are errors, correct the erroneous configuration and test again.

Reload or restart the OpenSSH server daemon with sudo systemctl [reload|restart] sshd.service.

Optionally, verify the service is running with sudo systemctl status sshd.service.

● sshd.service - OpenSSH server daemon
Loaded: loaded (/usr/lib/systemd/system/sshd.service; enabled; preset: enabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: active (running) since Tue 2024-04-02 21:05:08 EDT; 1h 14min ago
Docs: man:sshd(8)
man:sshd_config(5)
Process: 88931 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
Main PID: 88580 (sshd)
Tasks: 1 (limit: 4643)
Memory: 5.3M
CPU: 6.249s
CGroup: /system.slice/sshd.service
└─88580 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

VMware SDDC Manager

Optionally, back in the SDDC Manager SSH session, start a tail (tail -f /var/log/secure) of the /var/log/secure file before heading back to SDDC Manager GUI.

On the SDDC Manager GUI, configure the backup again. My SFTP server is
configured with the following:

IP: 192.168.92.4
User name: sftpuser
Password: VMware1!
Configured SFTP location on server: /var/sftp/sftpdata

I used the above information and used the following to complete the Site Settings
configuration. Pay attention with regards to the Backup Directory, ensure there is a trailing forward slash!
Backup Directory: /sftpdata/backup/
SSH Fingerprint: Populates automatically after entering the Host FQDN or IP
Encryption Passphrase: VMware1!VMware1!

This time, I received a banner that it was configuring the components!

When reviewing the tail of the log, we see that multiple sessions open and close. This is for the various components of VMware SDDC Manager and VMware NSX Manager.

Verify the backup actually wrote something on the server.

Conclusion

Wow, this was something else! I most certainly spent way too much time on figuring this out. I really hope that this will save you countless hours (or days) if you are also experiencing this.

---