VMware Cloud Foundation: Install custom SSL Certificates with XCA

If you set up a VMware Cloud Foundation (VCF) deployment you will notice all components (SDDC manager, vCenter, Platform Service Controllers, NSX manager & vRealize Log Insight) are using self-signed SSL certificates for their web services.
If you have a Microsoft Active Directory server or cluster you can use their Certificate Authority (CA) functionality to generate trusted certificates as described in the official documentation.
However there is an alternative if you are not willing to setup Microsoft servers or pay their license fees. You can create your own certificates by your internally trusted CA and let SDDC manager do the work of distributing them among the various VCF components.

In this example based on the corresponding documentation page I will use the freeware software XCA, which is a graphical frontend to create and manage X.509 certificates. It is available for Windows, macOS and Linux.

When you have downloaded and installed the software and are opening it for the first time, you need to create a new database (see “File” menu) as a starting point. It will ask you for a filename and a password which you need to enter each time you are accessing the database. Also you should set the default hash algorithm to “SHA256” in the options menu, as “SHA 1” is deprecated.

In the simplest case you would create a CA by hitting “New Certificate”, selecting the “CA” template (followed by “Apply all”), giving it at least a name (Internal name, commonName) and generating a private key for it.

In my case however I already had a CA up and running elsewhere, which I used to create an intermediate CA called “xca”. To be able to use that to create certificates in the XCA tool I first had to import the already created private key:

Then I imported both the certificates of the root and intermediate CA:

If you are not using a self created CA as described above you need to select the externally created root CA and click on “Trust” in the context menu: (the intermediate CA is then trusted automatically)

Now it was time to generate the certificate signing requests using the SDDC manager interface. Select all resources you want their certificates to replace and click on the “Generate CSR” button: (found under the “Security” tab of your workload/management domain)

This will let you download an tar.gz archive named like your workload domain. So for the management domain it is called “MGMT.tar.gz”. Extract that archive with your favorite tool, e.g. using “tar -xzf MGMT.tar.gz” for *nix. For Windows desktops 7-Zip is working fine, although you might need to extract in two steps (.tar.gz -> .tar -> extract contents).

After extraction you should have a folder also named like your workload domain with sub-directories named like the hostnames of your VCF components, containing a .csr file each. Import those in the “Certificate signing requests” tab in XCA using the “Import” button:

Pick a CSR, open the context menu and click on “Sign”:

The following window will appear. Make sure that the correct root or intermediate CA is selected for “Use this Certificate for signing”, and that a supported hash algorithm like “SHA 256” is selected: (ignore the Template selection)

In the next tab you can enter the time range the certificate will be valid. After entering a number you need to hit the “Apply” button. As all other important settings are already filled out from the CSR no further modifications are needed. Maybe the “X509v3 Subject Alternative Name” (SAN) field would be a good idea to fill out with the respective FQDNs and IP addresses (I will explain later on why).

After having the signing procedure repeated for all CSRs the “Certificate” tab of XCA should look like the next screenshot.
Here you need to export the created certificates to the same folders you imported the CSRs from with the same filename (with file extension “.crt”). Also make sure the export format is set to “PEM”:

You also need to export a certificate chain of the trusted CAs to file called “rootca.crt” placed in the extracted directory where the other sub-directories are located. This can be done with XCA as shown below:

For the SDDC manager to be able to import the certificate structure (including the previously exported CSRs) the folder structure needs to be in an tar.gz archive once again. You might need to delete the old archive downloaded previously as the same name is used.
In *nix use “tar -czf MGMT.tar.gz MGMT/”. Using 7-Zip it is again a two step procedure. First add the folder to a tar archive like this:

Then add the tar archive to a gzip archive using the default settings:

The resulting tar.gz file can then be uploaded in the menu opening after clicking on “Upload and install”:

If everything is done correctly the result should look like this:

All services except for the SDDC manager are restarted automatically, but you may need to close browser sessions if you still have old ones open or even clean your browsing cache.
If you do not want to reboot your SDDC manager use SSH and the “vcf” user to log into it and run the following commands:

su
sh /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh 

Of course you still need to import your locally created CA into the trusted folder of your browser of choice so that it show as “valid” HTTPS. This howto should help to accomplish this. In the end it should look like this:

One issue I found was that the connection between vCenter and NSX manager was no longer working with the new certificates. Searching the symptoms (vCenter displaying “No NSX Managers available. Verify current user has role assigned on NSX Manager.”) in the VMware knowledge base led me check the lookup/registration page of the NSX manager appliance. It appears that Cloud Foundation sets up both URLs using their IP addresses. After changing both to the PSC/vCenter FQDNs (as shown in below screenshot) and restarting the VMs everything was working again:

Another solution to solve this could be to add the IP addresses of each VCF
components into the individual “SAN” field when creating the certificates, as described above, so that the HTTPS connection is trusted in both ways.