FIPS 140-2 on VMware Cloud Builder

Overview

I went to deploy a VMware Cloud Builder appliance the other day with FIPS enabled and it turned out to be frustrating. Turns out, how the appliance is deployed matters. This post would not be possible without an extremely diligent, intelligent, and hard working colleague of mine, Vinny Santa Maria.

The how in this case is whether the appliance is deployed to ESXi directly or using vCenter. Hint? Deploying using vCenter seems to work correctly. Well, that’s great if you already have a virtualized environment and access to a vCenter. In a greenfield deployment, that is an awful lot of overhead and may not even be possible in the case of a purpose built and resource constrained infrastructure.

Want to skip to the fix? Click here.

Symptoms and Observations

The first indicator is when trying to upload the configuration file.

Internal Server Error, make sure the xlsx/json file is valid

---

An additional indicator is at the top of the screen. I actually never noticed it since most Broadcom applications use some indicator at the top of the screen and it is just ignored by me at this point. Another colleague of mine pointed this out.

The bringup services are currently not running or restarting. Please try again later.

---

When logged on to the command line of the appliance, check the status of the services with systemctl status and notice there are 5 queued services. In my observations, these will never start.

---

vcf-bringup-fips Service

The vcf-bringup-fips service is actually what checks if FIPS is enabled or not.

There is a system process located at /proc/sys/crypto/fips_enabled that when you cat out should return a 1 on a FIPS enabled appliance.

When checking the status of the service, this is how it should look.

root@cloud-builder [ /home/admin ]# systemctl status vcf-bringup-fips.service 
● vcf-bringup-fips.service - VMware Cloud Foundation Bringup FIPS Configuration
     Loaded: loaded (/etc/systemd/system/vcf-bringup-fips.service; disabled; vendor preset: enabled)
     Active: active (exited) since Tue 2024-12-17 16:35:53 UTC; 18h ago
    Process: 955 ExecStart=/bin/bash -c while ! /opt/vmware/bringup/scripts/fips/init-fips.sh; do sleep 5; done (code=exited, status=0/SUCCESS)
   Main PID: 955 (code=exited, status=0/SUCCESS)

Dec 17 16:35:53 cloudb bash[1074]: +++ head -1 /proc/sys/crypto/fips_enabled
Dec 17 16:35:53 cloudb bash[1064]: ++ os_fips=1
Dec 17 16:35:53 cloudb bash[1064]: ++ test 1 = 1
Dec 17 16:35:53 cloudb bash[1064]: ++ echo true
Dec 17 16:35:53 cloudb bash[958]: + fips_mode_initialized_and_matches=true
Dec 17 16:35:53 cloudb bash[958]: + echo 'fips_mode_initialized_and_matches? true'
Dec 17 16:35:53 cloudb bash[958]: fips_mode_initialized_and_matches? true
Dec 17 16:35:53 cloudb bash[958]: + test true = true
Dec 17 16:35:53 cloudb bash[958]: + exit 0
Dec 17 16:35:53 cloudb systemd[1]: Finished VMware Cloud Foundation Bringup FIPS Configuration.

However, when deploying directly to ESXi, the service looks like this.

root@cloud-builder [ /home/admin ]# systemctl status vcf-bringup-fips.service 
● vcf-bringup-fips.service - VMware Cloud Foundation Bringup FIPS Configuration
     Loaded: loaded (/etc/systemd/system/vcf-bringup-fips.service; disabled; vendor preset: enabled)
     Active: activating (start) since Wed 2024-12-18 10:26:52 UTC; 7min ago
   Main PID: 963 (bash)
      Tasks: 2 (limit: 4668)
     Memory: 7.4M
     CGroup: /system.slice/vcf-bringup-fips.service
             ├─ 963 /bin/bash -c while ! /opt/vmware/bringup/scripts/fips/init-fips.sh; do sleep 5; done
             └─3352 sleep 5

Dec 18 10:34:43 cloud-builder bash[3350]: +++ '[' -f /proc/sys/crypto/fips_enabled ']'
Dec 18 10:34:43 cloud-builder bash[3350]: +++ head -1 /proc/sys/crypto/fips_enabled
Dec 18 10:34:43 cloud-builder bash[3347]: ++ os_fips=1
Dec 18 10:34:43 cloud-builder bash[3347]: ++ test 0 = 1
Dec 18 10:34:43 cloud-builder bash[3347]: ++ echo false
Dec 18 10:34:43 cloud-builder bash[3323]: + fips_mode_initialized_and_matches=false
Dec 18 10:34:43 cloud-builder bash[3323]: + echo 'fips_mode_initialized_and_matches? false'
Dec 18 10:34:43 cloud-builder bash[3323]: fips_mode_initialized_and_matches? false
Dec 18 10:34:43 cloud-builder bash[3323]: + test false = true
Dec 18 10:34:43 cloud-builder bash[3323]: + exit 1

Remember the service is only checking if FIPS is enabled by testing the process listed above. On the ESXi deployed appliance, the test is 0 = 1 which will never test true.

Since the appliance is deployed from an OVA file, certain properties must be injected at deployment. The values can be checked using the ovfenv utility.

/opt/vmware/bin/ovfenv --key FIPS_ENABLE

On a vCenter deployed appliance, the returned value is True.

root@cloud-builder [ ~ ]# /opt/vmware/bin/ovfenv --key FIPS_ENABLE
True

On an ESXi deployed appliance, the returned value is true.

root@cloud-builder [ ~ ]# /opt/vmware/bin/ovfenv --key FIPS_ENABLE
true

The case of the returned values are different.

---

There is a script (/opt/vmware/bringup/scripts/fips/init-fips.sh) that tests APP_FIPS_MODE and compares it to OS_FIPS_MODE. The APP_FIPS_MODE value is set when deploying the appliance, it comes from the OVA deployment properties (/opt/vmware/bin/ovfenv). OS_FIPS_MODE is the current state of the appliance (/proc/sys/crypto/fips_enabled).

Here is the app_fips_mode function in the script.

function app_fips_mode() {
  # file exists?
  if [[ -f ${APP_FIPS_MODE} ]]; then
      if [[ `${APP_FIPS_MODE} -k FIPS_ENABLE` == 'True' ]]
      then
        echo "1"
      else
        echo "0"
      fi
  else
    # In Photon 5,
    # /opt/vmware/bin/ovfenv path does not exist and
    # FIPS will only be enabled at both OS and app level.
    if (( $(echo "$PHOTON_VERSION >= 5.0" | bc -l) )); then
        echo "1"
    else
        echo "${APP_FIPS_MODE} does not exist"
    fi
  fi
}

The Fix(es)

Fix 1

In the /opt/vmware/bringup/scripts/fips/init-fips.sh script, we need to add an additional test to account for either situation.

In the app_fips_mode function, change the if statement to test either a True case or a true case.

if [[ `${APP_FIPS_MODE} -k FIPS_ENABLE` == 'True' ]] || [[ `${APP_FIPS_MODE} -k FIPS_ENABLE` == 'true' ]]

Save the file and reboot the appliance.

---

Fix 2

The difference in how the appliance is deployed (vCenter vs direct to ESXi) will set the ovfEnv property with a different case (True for vCenter deployed and true for ESXi deployed). The thing that stays the same is the test on the appliance (testing for the True case). For this fix, alter the FIPS_ENABLE property in the /opt/vmware/etc/vami/ovfEnv.xml file.

First, copy the original file.

cp /opt/vmware/etc/vami/ovfEnv.xml /opt/vmware/etc/vami/ovfEnv.xml.orig

The file does not contain spaces so it is challenging to edit. The following command will make another copy to edit in an easy to read format.

xmllint --format /opt/vmware/etc/vami/ovfEnv.xml > /opt/vmware/etc/vami/ovfEnv.xml.new

Edit the file.

vi /opt/vmware/etc/vami/ovfEnv.xml.new

This is the incorrect value.

<Property oe:key="FIPS_ENABLE" oe:value="true"/>

It needs to be changed to the following.

<Property oe:key="FIPS_ENABLE" oe:value="True"/>

Save and copy the .new file to the original. The file size should grow to 1165 bytes with the extra spacing. The spaces will not affect the contents.

cp /opt/vmware/etc/vami/ovfEnv.xml.new /opt/vmware/etc/vami/ovfEnv.xml

systemctl restart vcf-bringup-fips.service

systemctl status vcf-bringup-fips.service

---