Creating workload domains in VMware Cloud Foundation 3.7.2 with NSX-T and vSAN

In VMware Cloud Foundation (VCF) workloads usually are deployed in one or more dedicated virtual infrastructure (VI) workload domains. During the VCF deployment (as shown in my earlier posts) the management workload domain (MWLD) is created with a minimum of four hosts. The WLD contains among other components the management vCenter and the SDDC manager.
For each VI workload domain (WLD) created using the SDDC manager a separate vCenter is deployed in the MWLD. The vCenters manage the WLD’s hosts and use the vSphere linked mode. As only fifteen vCenters can be linked as per current configuration maximums, currently up to 14 WLDs are supported.
Before the SDDC manager can create a WLD enough hosts (minimum three per WLD) need to be commissioned. Click on the button “Commission hosts” either in the Dashboard or the Inventory/Hosts view:

VMware Cloud Foundation – SDDC Manager, Commission hosts

The hosts need be be prepared similarly to the VCF deployment. This includes ESXi version, hardware configuration and network settings (e.g. DNS resolution) and shown in below checklist. In a later post I will provide some helpful PowerCLI snippets to accelerate the host preparation.

VMware Cloud Foundation – SDDC Manager, Commission hosts, Checklist

After clicking on “Proceed” the details of the hosts need to be provided. Either add each individual host manually (Select “Add new”) or perform a bulk commission by preparing and uploading a JSON file:

VMware Cloud Foundation – SDDC Manager, Commission hosts, host addition

The JSON template provided looks like this:

{
    "hostsSpec": [
        {
            "hostfqdn": "Fully qual. domain name goes here",
            "username": "User Name goes here",
            "storageType": "VSAN/NFS",
            "password": "Password goes here",
            "networkPoolName": "Network Pool Name goes here"
        },
        {
            "hostfqdn": "Fully qual. domain name goes here",
            "username": "User Name goes here",
            "storageType": "VSAN/NFS",
            "password": "Password goes here",
            "networkPoolName": "Network Pool Name goes here"
        }
    ]
}

Not only the host’s details (FQDN, credentials) and the storage type (preferably vSAN) needs to be provided, but the network pool to be used. Later on also license keys are required. A total of three license keys for vSphere, vSAN and NSX should be entered in the “Administration/License” screen of the SDDC manager.
Network pools are created in the “Administration/Network settings” screen. In this case VLAN-IDs and subnet for vMotion and vSAN separate from the default pool (used by the MWLD) are used:

VMware Cloud Foundation – SDDC Manager, Network pools

After the hosts are commissioned they show up in the “Usassigned hosts” tab:

VMware Cloud Foundation – SDDC Manager, Inventory/Hosts

Click on a host to show its details, e.g. manufacturer, model and storage capacity:

VMware Cloud Foundation – SDDC Manager, Inventory/Hosts, Host details

To create a new WLD use the “+ workload domain” button in the inventory:

VMware Cloud Foundation – SDDC Manager, Workload Domains

Select your storage in the next dialog box. vSAN and NFS are fully supported out of the box (Fibre Channel can be added later on manually, but must be managed independently):

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 0

In the first step of the VI configuration wizard enter names for the WLD, the first cluster and the organization the domain is intended for:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 1

Then enter a free IP address in the management subnet, a FQDN configured in your DNS servers and root password for the WLD’s vCenter:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 2

The most interesting part if you are enthusiastic for VMware’s SDN portfolio is the networking screen, which allows you to choose between the legacy product NSX-V or the 2019 released NSX-T version 2.4.
In both cases FQDNs, IP addresses and root/admin password for the NSX managers must be entered, as well as a VLAN ID used for the overlay transport (VXLAN for NSX-V; Geneve for NSX-T):

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 3

If you selected vSAN as primary storage provider in the first step you need to enter the PFTT (primary failure to tolerate) parameter in step four. “One failure to tolerate” means each data set is replicated once, similar to RAID 1. This means that any of the three required hosts can fail at any point in time without data loss.
If you have at least five hosts you can select PFTT=2, which means data is replicated twice, so two hosts may fail simultaneously. This is only the default setting however. PFTT can be set for each object via storage profiles later on, too.

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 4

In the next steps select the hosts which shall be used for initial WLD creation. Further hosts can be added to the WLD later. The host selection screen previews the accumulated resources of the selected hosts:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 5

In the License step select the license keys entered before from the drop down menus. Each license should provide enough capacity for each product (e.g. enough CPU socket count) and not be expired:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 6

The last two screens show a review of all entered parameters and a preview of the component names which will be created:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 7
VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Step 8

After finishing the wizard the creation progress can be tracked in the Tasks view in the bottom of the SDDC manager. If you click on the task all of its subtasks and their status are shown below:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Subtasks 1
VMware Cloud Foundation – SDDC Manager, Workload Domains, Add VI WLD, Subtasks 2

After some time the WLD creation tasks should succeed:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Creating VI WLD succeeded

Open the overview of the newly created WLD under the “Inventory/Workload Domains” to show its status. The “Services” tab features links to the vCenter and the NSX-T manager GUIs:

VMware Cloud Foundation – SDDC Manager, Workload Domains, Details of WLD

After a host is removed from a workload domain or the entire WLD is deleted the hosts are found under the tab “Unassigned hosts” again, but their state shows “Need Cleanup”:

VMware Cloud Foundation – SDDC Manager, Inventory/Hosts, Decommissioning

First select the checkbox on the left of each host needing cleanup and click on the button “Decommission selected hosts”.

Then login into the SDDC manager using SSH (e.g. “ssh vcf@sddc-mgr.local”) and prepare a JSON file containing the hosts and their management credentials as follows:

[
   {
     "host1.local":
       {
         "root_user": "root",
         "root_password": "VMware123!"
       }
   },
   {
     "host2.local":
       {
         "root_user": "root",
         "root_password": "VMware123!"
       }
   }
 ] 

Now run the following commands found in the VCF documentation to commence the cleanup:

su 
cd /opt/vmware/sddc-support
./sos --cleanup-decommissioned-host /tmp/dirty_hosts.json
VMware Cloud Foundation – SDDC Manager, Host cleanup script

Afterwards however there is still the task of the network cleanup, which requires access to Direct Console User Interface (DCUI).
If the network cleanup is not performed you will be presented with errors as shown below when trying to re-commission the hosts:

VMware Cloud Foundation – SDDC Manager, Host addition of partly cleaned up hosts, Error 1
VMware Cloud Foundation – SDDC Manager, Host addition of partly cleaned up hosts, Error 2

When logging into the ESXi management GUI in your browser you can see the left over distributed virtual switch and its port groups from the previous WLD:

VMware ESXi, Network settings

Perform the network cleanup by logging into the DCUI with the root user and then select “Network Restore Options”:

VMware ESXi, DCUI, Network Restore Options

Then select “Restore Network Settings” option which resets any network settings and devices to the defaults:

VMware ESXi, DCUI, Network Restore Settings
VMware ESXi, DCUI, Network Restore Settings, Done

Re-configuration of management network settings like IP address, subnet mask, default gateway and VLAN is needed afterwards.
Now of the cleaned hosts are ready to be re-commissioned, which works as shown in the beginning of this post.

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.