Cleaning up after the VMware Cloud Foundation 3.5 update

When a VMware Cloud Foundation deployment was updated to the current version, as described previously, a few tasks should be done afterwards.
First the vSAN datastore disk format version might need an upgrade. To check this head to the “Configure” tab of your DC in vCenter and click on “vSAN /Disk Management”:


vCenter cluster overview after VMware Cloud Foundation Update 3.5

Of course you should run the pre-check by clicking on the right button. If everything is working as it should it would look like this:

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN upgrade pre-check)

Now you can click the “Upgrade” button, which informs you this can take a while. Also you should backup your data/VMs elsewhere, especially if you select “Allow Reduced Redundancy”, which speeds up the process:

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN upgrade)

As you can see now the disk format version has changed from “5” to “7”:

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN upgraded)

However still some vSAN issues are displayed:

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN issues)

As this deployment is a “dark site”, meaning no internet access is available, the HCL database and Release catalog have to be updated manually.

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN Update)

The URL to download the 14.7 MB file can be found in a post from William Lam from 2015 or in this KB article. The release catalog’s URL is taken from another KB article. This file is less than 8 KB in size.
After uploading both using the corresponding “Update from file” buttons the screen should look like this:

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN updated)

The last remaining issue in this case was the firmware version of the host bus adapter connecting the vSAN datastore devices could not be retrieved (“N/A”):

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN Health)

Since the firmware version listed in the hosts iDRAC (see next screenshot) matches one of the “Recommended firmwares” from above I decided to rather hit “Silence alert”. Eventually one could look for an updated VIB file allowing the ESXi host to retrieve the firmware version from the controller.

iDRAC overview of storage controllers

One more effect of the upgrade from 3.0.1.1 to 3.5 is the appearance of three more VMs in vCenter. These are the old (6.5.x) instances of the platform service controllers and the vCenter. New instances with version 6.7.x have been deployed during the upgrade. After all settings had been imported from the old ones, these were apparently powered off and kept in case something would have gone wrong.
After a period of time and confirming everything works as expected those three VMs may be deleted from the datastore:

vCenter VM overview showing old PSCs and vCenter instances

Reusing storage devices for vSAN

Sometimes when a storage device (i.e. SSD or HDD) has been used for a previous vSAN deployment or has other leftovers it cannot be re-used (either for vSAN or a local VMFS datastore) right away. When you try to format the drive as shown below the error message “Cannot change the host configuration”:

Erase paritions highlighted in the Storage Devices view of a ESXi host in vSphere Client 6.7U1

The easiest way is to change the partition scheme from GPT to MSDOS via CLI (and back via GUI) and has been described in the community before.

However, even that may fail, e.g. because of the error “Read-only file system during write”. This can occur if the ESXi hypervisor finds traces of old vSAN deployments on the drive and refuses to overwrite these. In that case you first have to delete those traces manually. Log into the host in question as the root user and issue the vSAN commands needed. For deleting a SSD the command looks like this:

 vsan storage remove -s naa.6006016045502500c20a2b3ccecfe011 

Afterwards repeat the steps described in the link above to correctly (re)claim the entire diskspace and then use it according to your plan.