Creating workload domains in VMware Cloud Foundation 3.7.2 with NSX-T and vSAN

In VMware Cloud Foundation (VCF) workloads usually are deployed in one or more dedicated virtual infrastructure (VI) workload domains. During the VCF deployment (as shown in my earlier posts) the management workload domain (MWLD) is created with a minimum of four hosts. The WLD contains among other components the management vCenter and the SDDC manager.
For each VI workload domain (WLD) created using the SDDC manager a separate vCenter is deployed in the MWLD. The vCenters manage the WLD’s hosts and use the vSphere linked mode. As only fifteen vCenters can be linked as per current configuration maximums, currently up to 14 WLDs are supported.
Before the SDDC manager can create a WLD enough hosts (minimum three per WLD) need to be commissioned. Click on the button “Commission hosts” either in the Dashboard or the Inventory/Hosts view:

VMware Cloud Foundation – SDDC Manager, Commission hosts

The hosts need be be prepared similarly to the VCF deployment. This includes ESXi version, hardware configuration and network settings (e.g. DNS resolution) and shown in below checklist. In a later post I will provide some helpful PowerCLI snippets to accelerate the host preparation.

VMware Cloud Foundation – SDDC Manager, Commission hosts, Checklist

After clicking on “Proceed” the details of the hosts need to be provided. Either add each individual host manually (Select “Add new”) or perform a bulk commission by preparing and uploading a JSON file:

VMware Cloud Foundation – SDDC Manager, Commission hosts, host addition

The JSON template provided looks like this:

    "hostsSpec": [
            "hostfqdn": "Fully qual. domain name goes here",
            "username": "User Name goes here",
            "storageType": "VSAN/NFS",
            "password": "Password goes here",
            "networkPoolName": "Network Pool Name goes here"
            "hostfqdn": "Fully qual. domain name goes here",
            "username": "User Name goes here",
            "storageType": "VSAN/NFS",
            "password": "Password goes here",
            "networkPoolName": "Network Pool Name goes here"

Not only the host’s details (FQDN, credentials) and the storage type (preferably vSAN) needs to be provided, but the network pool to be used. Later on also license keys are required. A total of three license keys for vSphere, vSAN and NSX should be entered in the “Administration/License” screen of the SDDC manager.
Network pools are created in the “Administration/Network settings” screen. In this case VLAN-IDs and subnet for vMotion and vSAN separate from the default pool (used by the MWLD) are used:

VMware Cloud Foundation – SDDC Manager, Network pools

After the hosts are commissioned they show up in the “Usassigned hosts” tab:

VMware Cloud Foundation – SDDC Manager, Inventory/Hosts

Click on a host to show its details, e.g. manufacturer, model and storage capacity:

VMware Cloud Foundation – SDDC Manager, Inventory/Hosts, Host details

To create a new WLD use the “+ workload domain” button in the inventory:

VMware Cloud Foundation – SDDC Manager, Workload Domains

Select your storage in the next dialog box. vSAN and NFS are fully supported out of the box (Fibre Channel can be added later on manually, but must be managed independently):

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 0

In the first step of the VI configuration wizard enter names for the WLD, the first cluster and the organization the domain is intended for:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 1

Then enter a free IP address in the management subnet, a FQDN configured in your DNS servers and root password for the WLD’s vCenter:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 2

The most interesting part if you are enthusiastic for VMware’s SDN portfolio is the networking screen, which allows you to choose between the legacy product NSX-V or the 2019 released NSX-T version 2.4.
In both cases FQDNs, IP addresses and root/admin password for the NSX managers must be entered, as well as a VLAN ID used for the overlay transport (VXLAN for NSX-V; Geneve for NSX-T):

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 3

If you selected vSAN as primary storage provider in the first step you need to enter the PFTT (primary failure to tolerate) parameter in step four. “One failure to tolerate” means each data set is replicated once, similar to RAID 1. This means that any of the three required hosts can fail at any point in time without data loss.
If you have at least five hosts you can select PFTT=2, which means data is replicated twice, so two hosts may fail simultaneously. This is only the default setting however. PFTT can be set for each object via storage profiles later on, too.

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 4

In the next steps select the hosts which shall be used for initial WLD creation. Further hosts can be added to the WLD later. The host selection screen previews the accumulated resources of the selected hosts:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 5

In the License step select the license keys entered before from the drop down menus. Each license should provide enough capacity for each product (e.g. enough CPU socket count) and not be expired:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 6

The last two screens show a review of all entered parameters and a preview of the component names which will be created:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 7
VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 8

After finishing the wizard the creation progress can be tracked in the Tasks view in the bottom of the SDDC manager. If you click on the task all of its subtasks and their status are shown below:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Subtasks 1
VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Subtasks 2

After some time the WLD creation tasks should succeed:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Creating VI WLD succeeded

Open the overview of the newly created WLD under the “Inventory/Workload Domains” to show its status. The “Services” tab features links to the vCenter and the NSX-T manager GUIs:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Details of WLD

After a host is removed from a workload domain or the entire WLD is deleted the hosts are found under the tab “Unassigned hosts” again, but their state shows “Need Cleanup”:

VMware Cloud Foundation – SDDC Manager, Inventory/Hosts, Decommissioning

First select the checkbox on the left of each host needing cleanup and click on the button “Decommission selected hosts”.

Then login into the SDDC manager using SSH (e.g. “ssh vcf@sddc-mgr.local”) and prepare a JSON file containing the hosts and their management credentials as follows:

         "root_user": "root",
         "root_password": "VMware123!"
         "root_user": "root",
         "root_password": "VMware123!"

Now run the following commands found in the VCF documentation to commence the cleanup:

cd /opt/vmware/sddc-support
./sos --cleanup-decommissioned-host /tmp/dirty_hosts.json
VMware Cloud Foundation – SDDC Manager, Host cleanup script

Afterwards however there is still the task of the network cleanup, which requires access to Direct Console User Interface (DCUI).
If the network cleanup is not performed you will be presented with errors as shown below when trying to re-commission the hosts:

VMware Cloud Foundation – SDDC Manager, Host addition of partly cleaned up hosts, Error 1
VMware Cloud Foundation – SDDC Manager, Host addition of partly cleaned up hosts, Error 2

When logging into the ESXi management GUI in your browser you can see the left over distributed virtual switch and its port groups from the previous WLD:

VMware ESXi, Network settings

Perform the network cleanup by logging into the DCUI with the root user and then select “Network Restore Options”:

VMware ESXi, DCUI, Network Restore Options

Then select “Restore Network Settings” option which resets any network settings and devices to the defaults:

VMware ESXi, DCUI, Network Restore Settings
VMware ESXi, DCUI, Network Restore Settings, Done

Re-configuration of management network settings like IP address, subnet mask, default gateway and VLAN is needed afterwards.
Now of the cleaned hosts are ready to be re-commissioned, which works as shown in the beginning of this post.

Updating VMware Cloud Foundation dark site deployment from 3.5.1 to 3.7

Two weeks ago the latest and greatest in VMware’s SDDC came out: VMware Cloud Foundation (VCF) version 3.7.
Apart from including the current security patches (e.g. ESXi 6.7 EP 06 / build number 11675023) a couple of new automation features have been added, as you can see in the release notes.
Also the cloud builder for setting up greenfield Cloud Foundation and VMware Validated Design deployments have been merged. As you can see in my tweet from a while back this used to be to separate OVA files.

VCF 3.7 can be installed as a new deployment or upgraded from the previous version (3.5.1), which is what I did to a dark site I am maintaining.
The process is the same as in my previous posts:

VMware Cloud Foundation update version 3.7 – Bundle download

After downloading the bundle files (around 21 GB) on a PC with internet access and importing those into the SDDC manager you can trigger the first phase of the update process, which is updating the SDDC manager itself:

VMware Cloud Foundation update version 3.7 – Update in Progress

This took less than 22 minutes on current Dell EMC hardware:

VMware Cloud Foundation update version 3.7 – Finished phase 1 update

The new build numbers are 12695026 / 12695044 (UI).

VMware Cloud Foundation update version 3.7 – Build numbers after phase 1 update

After triggering the next update phase the vCenter and PSC instances are bumped from build number 10244745 to 11726888, which is the most current security update available:

VMware Cloud Foundation update version 3.7 – Build numbers after phase 2 update

The last step is upgrading the ESXi hosts to build number 11675023 which was released on 01/17/2019. Only recently (03/28/2019) a more current security patch was released, which will presumably be included in one of the future VCF upgrades.

VMware Cloud Foundation update version 3.7 – Build numbers after final update

Having all VCF 3.7 patches installed is confirmed by the displayed text “There is no update available”.

VMware Cloud Foundation: Install custom SSL Certificates with XCA

If you set up a VMware Cloud Foundation (VCF) deployment you will notice all components (SDDC manager, vCenter, Platform Service Controllers, NSX manager & vRealize Log Insight) are using self-signed SSL certificates for their web services.
If you have a Microsoft Active Directory server or cluster you can use their Certificate Authority (CA) functionality to generate trusted certificates as described in the official documentation.
However there is an alternative if you are not willing to setup Microsoft servers or pay their license fees. You can create your own certificates by your internally trusted CA and let SDDC manager do the work of distributing them among the various VCF components.

In this example based on the corresponding documentation page I will use the freeware software XCA, which is a graphical frontend to create and manage X.509 certificates. It is available for Windows, macOS and Linux.

When you have downloaded and installed the software and are opening it for the first time, you need to create a new database (see “File” menu) as a starting point. It will ask you for a filename and a password which you need to enter each time you are accessing the database. Also you should set the default hash algorithm to “SHA256” in the options menu, as “SHA 1” is deprecated.

In the simplest case you would create a CA by hitting “New Certificate”, selecting the “CA” template (followed by “Apply all”), giving it at least a name (Internal name, commonName) and generating a private key for it.

In my case however I already had a CA up and running elsewhere, which I used to create an intermediate CA called “xca”. To be able to use that to create certificates in the XCA tool I first had to import the already created private key:

XCA – Private Keys

Then I imported both the certificates of the root and intermediate CA:

XCA – CA certifcates

If you are not using a self created CA as described above you need to select the externally created root CA and click on “Trust” in the context menu: (the intermediate CA is then trusted automatically)

XCA – Trust root CA

Now it was time to generate the certificate signing requests using the SDDC manager interface. Select all resources you want their certificates to replace and click on the “Generate CSR” button: (found under the “Security” tab of your workload/management domain)

VMware Cloud Foundation – Generate CSR

This will let you download an tar.gz archive named like your workload domain. So for the management domain it is called “MGMT.tar.gz”. Extract that archive with your favorite tool, e.g. using “tar -xzf MGMT.tar.gz” for *nix. For Windows desktops 7-Zip is working fine, although you might need to extract in two steps (.tar.gz -> .tar -> extract contents).

After extraction you should have a folder also named like your workload domain with sub-directories named like the hostnames of your VCF components, containing a .csr file each. Import those in the “Certificate signing requests” tab in XCA using the “Import” button:

XCA – Import CSRs

Pick a CSR, open the context menu and click on “Sign”:

XCA – Sign CSR

The following window will appear. Make sure that the correct root or intermediate CA is selected for “Use this Certificate for signing”, and that a supported hash algorithm like “SHA 256” is selected: (ignore the Template selection)

XCA – Generate certificate – Source

In the next tab you can enter the time range the certificate will be valid. After entering a number you need to hit the “Apply” button. As all other important settings are already filled out from the CSR no further modifications are needed. Maybe the “X509v3 Subject Alternative Name” (SAN) field would be a good idea to fill out with the respective FQDNs and IP addresses (I will explain later on why).

XCA – Generate certificate – Extensions

After having the signing procedure repeated for all CSRs the “Certificate” tab of XCA should look like the next screenshot.
Here you need to export the created certificates to the same folders you imported the CSRs from with the same filename (with file extension “.crt”). Also make sure the export format is set to “PEM”:

XCA . Export signed certificates to individual sub-directories

You also need to export a certificate chain of the trusted CAs to file called “rootca.crt” placed in the extracted directory where the other sub-directories are located. This can be done with XCA as shown below:

XCA – CA chain export

For the SDDC manager to be able to import the certificate structure (including the previously exported CSRs) the folder structure needs to be in an tar.gz archive once again. You might need to delete the old archive downloaded previously as the same name is used.
In *nix use “tar -czf MGMT.tar.gz MGMT/”. Using 7-Zip it is again a two step procedure. First add the folder to a tar archive like this:

7-Zip create tar archive

Then add the tar archive to a gzip archive using the default settings:

7-Zip – Create tar.gz archive

The resulting tar.gz file can then be uploaded in the menu opening after clicking on “Upload and install”:

VMware Cloud Foundation – Upload and Install Certificates

If everything is done correctly the result should look like this:

VMware Cloud Foundation – Upload and Install Certificates successful

All services except for the SDDC manager are restarted automatically, but you may need to close browser sessions if you still have old ones open or even clean your browsing cache.
If you do not want to reboot your SDDC manager use SSH and the “vcf” user to log into it and run the following commands:

sh /opt/vmware/vcf/operationsmanager/scripts/cli/

Of course you still need to import your locally created CA into the trusted folder of your browser of choice so that it show as “valid” HTTPS. This howto should help to accomplish this. In the end it should look like this:

VMware vCenter with secure HTTPS connection

One issue I found was that the connection between vCenter and NSX manager was no longer working with the new certificates. Searching the symptoms (vCenter displaying “No NSX Managers available. Verify current user has role assigned on NSX Manager.”) in the VMware knowledge base led me check the lookup/registration page of the NSX manager appliance. It appears that Cloud Foundation sets up both URLs using their IP addresses. After changing both to the PSC/vCenter FQDNs (as shown in below screenshot) and restarting the VMs everything was working again:

VMware NSX manager

Another solution to solve this could be to add the IP addresses of each VCF
components into the individual “SAN” field when creating the certificates, as described above, so that the HTTPS connection is trusted in both ways.

Updating VMware Cloud Foundation from 3.5 to 3.5.1

Business as usual to keep a dark site VCF deployment up to date…

  • Generating marker file on the SDDC manager
  • Using a workstation with internet access to download the update bundle(s) (in this case “bundle-8203.tar”, almost 7 GB big) and delta file
  • Uploading and importing them into the SDDC manager
  • Fixing file permissions
  • Install the update using the web interface of the SDDC manager:
VMware Cloud Foundation update version 3.5.1 – Import successful
VMware Cloud Foundation update version 3.5.1 – Update initialization successful

A countdown timer appears at the top of the SDDC manager admin page when you use the “Schedule Update” function:

VMware Cloud Foundation update version 3.5.1 – Update about to start
VMware Cloud Foundation update version 3.5.1 – Update in Progress
VMware Cloud Foundation update version 3.5.1 – Update Updated đź™‚

Everything went smoothly as expected in less than 21 minutes.

The release notes list what’s new:

  • Multi-Cluster NSX-T Support – Enables deployment of multiple clusters in a NSX-T based Workload Domain.
  • Custom ISO Support for Lifecyle Management – Enables customer-specified ISOs in place of VMware stock images for ESXi upgrades.
  • Miscellaneous Bug Fixes – Includes multiple bug fixes from the Cloud Foundation 3.5 release.

Cleaning up after the VMware Cloud Foundation 3.5 update

When a VMware Cloud Foundation deployment was updated to the current version, as described previously, a few tasks should be done afterwards.
First the vSAN datastore disk format version might need an upgrade. To check this head to the “Configure” tab of your DC in vCenter and click on “vSAN /Disk Management”:

vCenter cluster overview after VMware Cloud Foundation Update 3.5

Of course you should run the pre-check by clicking on the right button. If everything is working as it should it would look like this:

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN upgrade pre-check)

Now you can click the “Upgrade” button, which informs you this can take a while. Also you should backup your data/VMs elsewhere, especially if you select “Allow Reduced Redundancy”, which speeds up the process:

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN upgrade)

As you can see now the disk format version has changed from “5” to “7”:

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN upgraded)

However still some vSAN issues are displayed:

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN issues)

As this deployment is a “dark site”, meaning no internet access is available, the HCL database and Release catalog have to be updated manually.

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN Update)

The URL to download the 14.7 MB file can be found in a post from William Lam from 2015 or in this KB article. The release catalog’s URL is taken from another KB article. This file is less than 8 KB in size.
After uploading both using the corresponding “Update from file” buttons the screen should look like this:

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN updated)

The last remaining issue in this case was the firmware version of the host bus adapter connecting the vSAN datastore devices could not be retrieved (“N/A”):

vCenter cluster overview after VMware Cloud Foundation Update 3.5 (vSAN Health)

Since the firmware version listed in the hosts iDRAC (see next screenshot) matches one of the “Recommended firmwares” from above I decided to rather hit “Silence alert”. Eventually one could look for an updated VIB file allowing the ESXi host to retrieve the firmware version from the controller.

iDRAC overview of storage controllers

One more effect of the upgrade from to 3.5 is the appearance of three more VMs in vCenter. These are the old (6.5.x) instances of the platform service controllers and the vCenter. New instances with version 6.7.x have been deployed during the upgrade. After all settings had been imported from the old ones, these were apparently powered off and kept in case something would have gone wrong.
After a period of time and confirming everything works as expected those three VMs may be deleted from the datastore:

vCenter VM overview showing old PSCs and vCenter instances

Updating VMware Cloud Foundation from to 3.5

In this post I would like to show you the process of updating a VCF deployment at a customer site to the current version which was released in mid december.
The pictures show only the update of the management workload domain, as that is the only one currently available there. If you have multiple VI/VDI workload domain still you have to update the management domain first, and then individual workload domains.

The steps necessary are the same as in previous updates. E.g. if you are located at an environment isolated from the internet you can use a laptop to download the bundle files based on a delta file provided by the SDDC manager and import these afterwards, as described in one of my previous posts.
The update itself can be scheduled or started immediately. The process is the same as before, but consists of multiple phases.

The first phase updates the VCF services itself, including domain manager, SDDC manager UI and LCM:

VMware Cloud Foundation Update 3.5 phase 1

Afterwards the NSX components are updated to version 6.4.4 as shown in below screenshot:

VMware Cloud Foundation Update 3.5 phase 2

In the next phase the platform service controllers (for the management domain typically two) and the vCenter are updated to version 6.7. Sadly in the first release of the VCF 3.5 update bundles there was a but resulting in an error in the stage “vCenter upgrade create input spec”:

VMware Cloud Foundation Update 3.5 phase 3 (error)

The SDDC manager’s log file “/var/log/vmware/vcf/lcm/lcm-debug.log” only showed a “java.lang.NullPointerException
error at the component “com.vmware.evo.sddc.lcm.orch.PrimitiveService“, which didn’t help me much, so after a unlucky Google search I contacted VMware’s support. Upon opening a support case on a very friendly Senior Technical Support Engineer got back to me within minutes and pointed my attention to this knowledge base article. Apparently the issue cannot be fixed in place, but a new update bundle is available replacing the buggy one. If your SDDC manager has internet access it can download the bundle automatically, but if you are at a “dark site” you first need to get rid of the faulty bundle’s id by running the following python script:

/opt/vmware/vcf/lcm/lcm-app/bin/ 0864d5b4-2901-4198-83b8-457985af2ff8

Afterwards a new marker file has to be created and transferred to a workstation with internet access where the updated bundles are downloaded: (same procedure as described before)

VMware Cloud Foundation Update 3.5 phase 3 – Download fixed bundles

This screen shows the new successful import of the previously downloaded bundles after copying it back to the SDDC manager:

VMware Cloud Foundation Update 3.5 phase 3 – Imported fixed bundles

Finally we can retry the phase 3. As you can see here a new screen appears now:

VMware Cloud Foundation Update 3.5 phase 3, fixed – Step 1

As vCenter appliances can not be upgraded from 6.5 to 6.7 directly a new appliance has to be deployed which then imports all settings from the old one. To be able to complete this process the SDDC manager needs a temporary IP address for that new appliance in the same range as the vCenter/PSC:

VMware Cloud Foundation Update 3.5 phase 3, fixed – Step 2

Check the review screen to confirm the temporary IP settings and hit “Finish” to start the update:

VMware Cloud Foundation Update 3.5 phase 3, fixed – Step 3

Hooray! The update process did not fail at the stage it did before:

VMware Cloud Foundation Update 3.5 phase 3, fixed – Update in progress

After a little more than an hour all three appliances are up-to-date:

VMware Cloud Foundation Update 3.5 phase 3, fixed – Finished

As we can see in the overview screen of the domain all components are updated, except for the ESXi hosts:

VMware Cloud Foundation Update 3.5 – Version overview after phase 3

This means the fourth and final phase can be started: the update of the ESXi hosts to build 1076412:

VMware Cloud Foundation Update 3.5 – Phase 4

This concludes the update to VCF 3.5 as all components now have the current build numbers:

VMware Cloud Foundation Update 3.5 – Version overview after phase 4

The next screenshot of the Update history section shows the update from 3.0.1 to and the four updates from above:

VMware Cloud Foundation – Update history

Deploying VMware Cloud Foundation 3.0.1 on EoL servers

In my company’s lab I found a couple of quite old x86 servers, which were not in use anymore. The rack servers are in fact so old, that the original manufacturer (Sun) doesn’t exist anymore. The model is named “X4270 M2” and labeled end-of-life by Oracle for a while now. They are equipped with Intel Xeon processors released in 2011 (!), code name Westmere EP. That is in fact the oldest dual socket CPU generation by Intel which is supported by ESXi 6.7 (needed soon for VCF 3.5 upgrade).
I found some more servers, but those are equipped with Nehalem CPUs, so not hypervisor material; One possibility to give them a new purpose could be as a baremetal NSX-T edge

The main concerns whether VCF could be successfully deployed on old hardware
like that (when vSAN Ready Nodes, as required by VCF, were not even a thing yet) were compatibility with VMware’s HCL (especially HDDs, SSDs & raid controller), lack of 10 GbE adapters and not enough RAM.
Preparing the five servers (four for the management domain and another for the Cloud Builder VM) with ESXi was by the book, except for a well known workaround needed on old Sun servers.
For NTP, DNS and DHCP the OPNsense distribution was used once more.
After uploading the filled-out Deployment Parameter Sheet the Cloud Builder VM started its validation, resulting only in one warning/error regarding cache/capacity tier ratio which can be acknowledged. In fact the same message was displayed at a customer´s site with Dell PowerEdge R640 nodes with 4TB/800GB SSDs. This seems to be related with a known issue

VCF Configuration File Validation

However after hitting Retry another error was displayed saying that no SSDs available for vSAN were found. This could be confirmed when logging into any of the hosts ESXi interface.
The Intel SSDs were marked as hard disks and could not be marked as Flash via the GUI. The reason for this is the RAID controller by LSI which does not have a SATA bypass mode, meaning you have to create a RAID 0 virtual disk for each pass-through drive, so that the hypervisor has no clue about which hardware device lies underneath.
Upon investigating further in VMware´s KB a storage filter for local devices can be added via the CLI so that after a reboot that device will be marked correctly as SSD:

esxcli storage core device list
[Find the SSD which is supposed to be marked as such, e.g. "naa.600605b00411be5021404f8240529589"]
esxcli storage nmp satp rule add --satp=VMW_SATP_LOCAL --device naa.600605b00411be5021404f8240529589 --option "enable_local enable_ssd"

Finally the Cloud Foundation Bring-Up Process could be initiated.
Still no luck however, as an error deploying the NSX manager was displayed:

SDDC Bring Up

As the error was in that stage meant that the platform services controllers, SDDC controller and vCenter were already successfully deployed and reachable. After logging into the latter it was clear all VMs were put on the first host and so no more RAM space was available for the NSX manager. 

The first attempt to fix this problem was to migrate all VMs which were deployed so far onto the other three hosts. Afterwards the Bring Up process could be picked up by hitting Retry, but eventually the same error came up again.
It became apparent, that the four hosts were not equipped with a sufficient amount of RAM (24 GB) after all.
After  shutting down the hosts in the correct order more RAM was added (however still less than the amount described as required minimum: 72 GB vs. 192 GB) and then started up again.

Now the Bring Up went through, resulting in an up-to-date private cloud SDDC automated deployment on >8 year old hardware…

Of course this setup is only valid for lab tests as not respecting VMware’s minimum requirements and design recommendations is not supported and not  suited for production.

Manually downloading the VMware Cloud Foundation Update Bundle

If your VCF SDDC deployment does not have Internet connectivity you can manually download update bundles on another machine and import it afterwards.
Here are the necessary steps on a Windows workstation.

Password fields for SDDC Manager “vcf” and “root” users.

First use Putty to connect to the SDDC manager as user “vcf” and the password set in the cloud foundation deployment parameter spreadsheet (red circle in the image above) and run the following commands:

cd /opt/vmware/vcf/lcm/
[enter root password; see green circle in top image]
mkdir bundleimport
chown vcf:vcf bundleimport
cd lcm-tools/bin/
./lcm-bundle-transfer-util --generateMarker

Create a folder on your windows machine (e.g. “C:\…\bundleupdate”) and copy the remote files “markerFile” and “markerFile.md5” from “/home/vcf/”, as well as the entire “/opt/vmware/vcf/lcm/lcm-tools/” directory structure using WinSCP. In that folder create another subfolder; In my case I called it “downloadedBundles”.
Make sure you have a current version of Java (JRE) installed.
Open a command prompt and run the following the commands: (when asked enter your password)

cd C:\...\bundleupdate
./lcm-bundle-transfer-util -download -outputDirectory C:\...\
bundleupdate\downloadedBundles -depotUser {your username} -markerFile C:\...\bundleupdate\markerFile
-markerMd5File C:\...\bundleupdate\markerFile.md5
WinSCP (above) and lcm-bundle-transfer-util (below)

After the download is completed unplug your internet cord and connect to your VCF deployment once more. Using WinSCP copy the content of your local folder “C:…\
bundleupdate\downloadedBundles” to “/opt/vmware/vcf/lcm/bundleimport”. Then use Putty again to run these commands:

cd /opt/vmware/vcf/lcm/lcm-tools/bin
chmod -R 777 ../../bundleimport
./lcm-bundle-transfer-util -upload -bundleDirectory /opt/vmware/vcf/lcm/bundleimport
Successfully imported update bundle listed in the Repository section.

Updating VMware Cloud Foundation 3.0.1 to

Here are a couple of screenshots of the SDDC Manager GUI showing the update process of a VCF deployment at a customer site (hostnames are edited out)…

Successfully downloaded update bundles listed in the Repository section.
Available updates for the Management Domain in the “Update/Patches” pane after completing the Precheck.
Starting the update by clicking “Update now” or “Schedule Update”.
Update in progress…
Detailed update steps in Progress / Queued.
ESX build number 10175896 shown in vCenter before update
ESX build number 1079125 shown in vCenter after update
The date on which the update completed is shown under “Update History”.

Keeping up with the Cloud Foundations

I am currently helping a customer build a infrastructure platform to run a couple of
virtualized applications. The decision to use VMware products was already made before I joined the project, but at that stage (middle of the year) it was still uncertain whether the deployment / networking would both be “old school” (setting up everything by hand / VLANs seperated by physical firewalls) or if new approaches should be applied.
My experience with NSX and some articles I read about a new way of deploying VMware based SDDCs, namely the VMware Cloud Foundation (VCF), layed out the foundation (see what I did there…) for our new private cloud.

After continuing to dive into the VCF stack and its ideas (this free fundamentals course is great for starters) it quickly became clear that this could help reduce resources spent on deploying and operating the project’s infrastructure
drastically and also prevent human errors, as entire batches of tasks are automated, following the VMware Validated Designs

While planning the environment the latest VMware Cloud Foundation version available was 2.3.2. For this version the hardware compatibility list (both compute and networking equipment) was rather short, so for hardware selection Dell components were chosen. Until some more workshops were conducted an the boxes finally arrived some time passed, so a lot happened in the mean time…

During the VMworld US 2018 the new version 3.0 was announced and was released shortly after. The big difference introduced in this mayor update was focusing on VMware’s own products. When pre-3.0 versions also included the networking stack, supporting only certain models from a handful of vendors (Cisco, Juniper, QCT, Dell), now any underlay network supporting 1600 byte MTUs and 10 Gbps ethernet and all vSAN Ready Nodes (> 20 vendors) meeting the required/supported minimums could be used, making even brown-field scenarios possible.

More than a test deployment of the 3.0 Cloud Builder VM to download the deployment parameter spreadsheet and prerequisite checklist didn’t see the light of the day in the project, as by the time the hardware was installed 3.0.1 was already available to download. This minor version jump featured some bug fixes and improvements. For example it was no longer necessary to convert the Excel spreadsheet containing the 
deployment parameters (IP addresses/networks, license details, passwords) into JSON format with the included Python script on your own. The 3.0.1 Cloud Builder VM web GUI accepts the Excel file directly. Very nice!

The entire VCF 3.0.1 deployment took less than two hours from uploading the parameter spreadsheet to finishing the bring up, leaving us with a ready to use environment with vCenter, two Platform Service Controllers, vSAN, NSX, vRealize Log Insight cluster and, of course, the new SDDC manager.
The preparation of our hosts (Dell PowerEdge vSAN ReadyNodes) with ESX 6.5 was pretty easy. For DHCP (VXLAN transport VLAN), DNS & NTP I set up a HA cluster of OPNsense gateways. Some pictures from the deployment process will follow in a separate post.

Shortly after this another new version came out ( As that only contains the current security patches for ESX 6.5 there only is a update bundle, not an OVA download.

Last week the next long awaited mayor release was published: 3.5. Again being available via upgrade or fresh OVA deployment it includes a log of changes. These were already announced at this year’s VMworld Europe, which I had the fortune to attend for the first time. Besides more bug fixes the jump to the current 6.7 releases of ESX, vCenter & vSAN is the biggest news (finally no need for Flash client – long live HTML5!), along with NSX 6.4.4 and updated version of vRLI, SDDC Manager and so on. Now also included is NSX-T 2.3.0, but only for workload domains – the management domain continues to rely on NSX(-V). This is supposed to pave the road for container based workloads like PKS/Kubernetes.

After the holidays I will continue the story with both results from upgrading the customer’s site to 3.5 and also deploying 3.5 at my company’s lab on older hardware, so stay tuned…