Deploying a vRealize Network Insight 4.2 Collector/Proxy to receive NetFlow data from OPNsense routers

vRealize Network Insight [vRNI] supports receiving and processing flow information from a variety of network equipment from different vendors out of the box, but also offers the possibility to ingest NetFlow/IPFix data from third party devices, e.g. physical routers.

Assuming you already have you vRNI instance deployed head to the Settings page of your vRNI WebGUI and click on “Acounts and Data Sources” to add such data sources.
If not you can deploy vRNI quickly using the vRealize Suite Lifecycle Manager as described in this older blog post. It shows an older version of vRNI (4.1), but the process is the same for 4.2.

vRealize Network Insight – Accounts and Data Sources

The button “Add Source” brings you to a list of all supported sources: (The option “Phyiscal Flow Collector” is only available if you have an Enterprise license registered in vRNI)

vRealize Network Insight – Accounts and Data Sources, Add Data Source

The minimum deployment of vRNI has a platform VM, which you use to administer and use the tool, and a collector VM (formerly called proxy), which can be selected as target for most data sources.
To be able to receive NetFlow data from a physical device however you need another dedicated collector VM. If this was not created earlier the screen informs you that no collector VM is available:

vRealize Network Insight – Accounts and Data Sources, Add Phyiscal Data Source, No Collector VM available

It does however offer you a button “Add Collector VM” to help create one.
When clicking the button a shared secret is displayed in a popup, which should be stored, as it is needed later on:

vRealize Network Insight – Accounts and Data Sources, Add Phyiscal Data Source, Add Collector VM

Download the “vRealize Network Insight – Proxy OVA file” (7 GB) from my.vmware.com and either deploy it via command line (see further below) or the vSphere WebGUI:

vSphere Client – Deploy OVF Template

Enter the shared secret from before in the step “Customize template”:

vSphere Client – Deploy OVF Template, Customize template

An alternative to deploying the OVA via the WebGUI is VMware’s OVF Tool, which allows you to deploy virtual appliance from the command line of your operating system (Windows, Linux or MacOS). The virtual appliances are distributed as file bundles, which usually contain the description (.ovf), the virtual disks (.vmdk in case of VMware environments) and a manifest (.mf) file containing hashes of the other files. For easier handling a tar archive with the file extentions .ova is created, containing these files.

To use the OVF Tool first download the current version (as of writing this post 4.3.0 U2) from VMware {code} and install it.
Then you can deploy the OVA directly to your vCenter with the following command: (modify datastore name, VM folder, VM name, port group name, download path, credentials, data center and cluster names according to your environment and enter the shared secret from before in the placeholder xxxxxx)

/Applications/VMware\ OVF\ Tool/ovftool -dm=thin -ds="vSAN xyz" --vmFolder="Management\ VMs" --acceptAllEulas --allowAllExtraConfig --name=vrni-collector2 --deploymentOption=large --net:"VM Network"="vRack-DPortGroup-vRealize" --prop:Proxy_Shared_Secret=xxxxxx  /home/user/Downloads/VMware-vRealize-Network-Insight-4.2.0.1562947515-proxy.ova vi://username:password@vcenter.rainpole.local/Datacenter/host/Cluster/

The prefix “/Applications/VMware\ OVF\ Tool” is only needed if you are running MacOS and did not add the directory where the OVF Tool was installed to to the $PATH environment variable.
Select one of the deployment options, depending on your expected system load:

Deployment Options:
                medium: vCPUs: 4, Memory: 12GB.
                large: vCPUs: 8, Memory: 16GB.
                extra_large: vCPUs: 8, Memory: 24GB.

After a while the deployment should succeed with the following messages:

Opening OVA source: VMware-vRealize-Network-Insight-4.2.0.1562947515- -proxy.ova
The manifest validates
Opening VI target: vi://username@vcenter.rainpole.local/Datacenter/host/Cluster/
Deploying to VI: vi://username@vcenter.rainpole.local/Datacenter/host/Cluster/
Transfer Completed
Completed successfully 

If you forgot to supply the shared secret as an argument you will receive the following error upon trying to power up the VM:

vSphere Client – Collector VM power on failed

You can still enter or, if entered false informarion earlier, correct the shared secret in the vApp Options properties as shown below:

vSphere Client – vApp properties of Collector VM

Upon clicking the edit button this popup allows adjusting the value:

vSphere Client – vApp properties of Collector VM, set value

After powering it up the appliance needs to be initially configured via the VM console. Login with the presented credentials (consoleuser / ark1nc0ns0l3) and enter “setup”:

vSphere Client – Collector VM setup start in VM console

Follow the wizard and enter the configuration options according to your environment:

vSphere Client – Collector VM setup finished in VM console

After finishing the configuration of the collector (formerly called proxy) you can select it from the drop-down list when adding a new physical netflow source at the “Accounts and Data Sources” page as shown in the beginning of the post. Don’t forget to give it a nickname: (e.g. the name of the collector VM or Netflow_collector)

vRealize Network Insight – Accounts and Data Sources, Add Phyiscal Data Source, Collector VM available

Now you can send NetFlow information from physical sources to port 2055 of the collector VMs IP address. NetFlow versions 5, 7, 9 and IPFIX are supported by vRNI, but keep in mind, that version 5 does not support IPv6.

To test the deployment I used the free open source firewall distribution OPNsense, based on FreeBSD.
As described in the OPNsense Wiki NetFlow destinations and capture details can the configured in the “Reporting” section:

OPNsense configuration, Reporting: NetFlow

After a while vRNI should have received some flows, visible in the “Accounts and Data Sources” page:

vRealize Network Insight – Accounts and Data Sources, Flow count

A quick test can be done with the following query suggested by Martijn Smit’s blog:

flow where Flow Type = 'Source is Physical' and Flow Type = 'Destination is Internet'

Further configuration of the NetFlow source or mapping in vRNI may be needed, e.g. regarding DNS or VLAN, which is both mentioned in Martijn Smit’s blog article.

Deploying vRealize Network Insight 4.1.0 with vRSLCM

In the beginning of May vRealize Network Insight 4.1 [vRNI] was released with a lot of interesting new features and enhancements described in the release notes.

It is getting more and more popular to use the vRealize Suite Lifecycle Manager appliance to deploy vRealize components like vRNI. In earlier posts I described how to deploy and update this tool to the current version as shown on below screenshot:

vRealize Suite Lifecycle Manager Version 2.1.0 Patch 1

In that version however support for vRNI 4.1.0 does not come out of the box. You rather have to install a product support package available in the VMware Marketplace / Solution Exchange first.

Download page for vRealize Network Insight 4.1.0 product support pack for vRealize Suite Lifecycle Manager

After installing the .pak file in the vRSLCM GUI under the “Settings/System Administration” page the new version needs to activated by clicking on the “Apply version” button:

vRealize Suite Lifecycle Manager – Installing a product support pack

You can check which products are supported by your deployment any time by clicking on the user name in the top right corner and then on “Products”, which opens up a pop up window.
The message “Policy successfully refreshed” confirms the new version is applied correctly:

vRealize Suite Lifecycle Manager – Applying a installed product support pack

Of course vRSLCM needs access to the product binaries. If the appliance has internet access and you would provide your my.vmware.com credentials it can download the .ova files directly.
For dark sites you can download both the “proxy” and “platform” .ova files on your workstation and upload them using SCP/SFTP: (screenshot shows WinSCP)

Uploading .ova files to vRealize Suite Lifecycle Manager using WinSCP

You need to add the product binaries to the product binary repository by entering the base location where you uploaded the .ova files earlier and then click on the “Discover” button. Finally select the added binaries and click “Add”:

vRealize Suite Lifecycle Manager – Adding product binaries

It takes a while until the product binaries are mapped and show up in the list:

vRealize Suite Lifecycle Manager – Adding product binaries in progress

Now you can deploy vRNI using vRSLCM by adding it to an existing environment or by creating a new environment. You have two deployment options for vRNI: Standard (1 Platform VM and 1 Cluster VM) or Cluster (3 Platform VMs and 1 Cluster VM). If you select “Cluster” only large nodes will be deployed, otherwise you can choose from “Standard” or “Large”.

This blog post shows all the required steps in between (prodiving certificate information, network details like IP addresses, subnet mask, gateway, portgroup and so on). Although the post is based on older versions of both vRealize Suite Lifecycle Manager and Network Insight the steps are mostly the same.

After entering all the details for creating a new environment you should run the pre-check validations:

vRealize Suite Lifecycle Manager – Pre-checks for deploying vRealize Network Insight in progress

If the validation succeeds you can commence the environment creation:.

vRealize Suite Lifecycle Manager – Pre-checks for deploying vRealize Network Insight successful

During the environment creation you can track the progress under the corresponding “In progress” request:

vRealize Suite Lifecycle Manager – Deploying vRealize Network Insight in progress

Once the request completes the deployment is ready to use:

vRealize Suite Lifecycle Manager – Deploying vRealize Network Insight successful

You can access the vRNI GUI via HTTPS on the configured address. Use the default admin user “admin@local” and the password you selected:

vRealize Network Insight login page

After first login the main features are explained in four separate screens:

vRealize Network Insight welcome page 1/4
vRealize Network Insight welcome page 2/4
vRealize Network Insight welcome page 3/4
vRealize Network Insight welcome page 4/4

You can use the self service wizard which helps you configure and learn about your vRNI deployment. Among the first steps it suggests to add data sources like vCenters and NSX managers:

vRealize Network Insight – Self Service

Apart from physical devices like routers and switches a whole variety of transport and infrastructure components can be added as data source:

vRealize Network Insight – Adding accounts and data sources

After some time to record flow information vRealize Network Insight is ready to display the first example path, in this case how a VM, which is attached to a logical switch (NSX-T 2.4 segment), connects to the Internet. The path from the T1 distributed router on the same host as the VM (cyan background) to the service router on the Edge Transport Node (purple background) is visible. As the physical switches and routers behind the NSX-T edges have not been configured as data source (yet) no further topology information is available between the service router and the Internet.

vRealize Network Insight – First packet flow/path

Installing vRealize Suite Lifecycle Manager 2.1.0 Patch 1

In the middle of May the first patch for the current version of everyone’s favorite tool to deploy and manage vRealize components was released.
In this KB the 12 issues resolved are listed.

To install it first download the patch file from my.vmware.com:

vRSLCM 2.1.0 Patch 1 download

Then open up “System Administration” page in the management GUI:

vRSLCM 2.1.0: System Administration

After clicking on the “Install Patch” button select the file download previously and wait for it being uploaded:

vRSLCM 2.1.0 Patch 1 Installation Step 1

Click on “Next” and review the details before finishing with the “Install” button:

vRSLCM 2.1.0 Patch 1 Installation Step 2

As with every update of a VMware product taking a snapshot and/or backing up the configuration before proceeding is recommended:

vRSLCM 2.1.0 Patch 1 Installation Step 3

After a short while the new build version is visible in the GUI:

vRSLCM 2.1.0 Patch 1 installed

Deploying two vRLI 4.7.1 clusters with vRealize Suite LCM 2.0 & setting up forwarding with SSL

After deploying vROPS using the vRSLCM yesterday, today the task was to deploy two separate instances of vRealize Log Insight. Both instances should consist of a cluster of one master and three workers (deployment type “Medium with HA”) and be placed on different hypervisor clusters, each managed by their own vCenter and separated by a third-party firewall. Finally the “outer” vRLI cluster would forward their received telemetry onto the “inner” cluster, which will function as part of a central SIEM platform.

The first step is to deploy both of the clusters. Again the “Create Environment” screen is used:

vRealize Suite Lifecycle Manager – Create Environment screen

After being finished with entering all the deployment parameters the pre-check is performed, but failed. Allegedly the IP addresses provided could not be resolved. Correctly configured Active Directory servers with the according A- and (reverse) PTR-entries were set up and reachable, so the warnings were ignored:

vRealize Suite Lifecycle Manager – Create Environment screen (Pre-check)

The environment creation is initiated:

vRealize Suite Lifecycle Manager – Create Environment screen (Initiated)

After deploying the master the three workers are deployed in parallel:

vRealize Suite Lifecycle Manager – Create Environment screen (In progress)

After deploying the three workers the LCM fails to configure the supplied NTP servers for some reason:

vRealize Suite Lifecycle Manager – Create Environment screen (Error)

At this point you have two options. The first one being deleting the environment (including the VMs by the below checkbox) and starting over: (e.g. if you actually made a mistake)

vRealize Suite Lifecycle Manager – Delete Environment screen

The other option is to resume the request: (The arrow on the right already disappeared after clicking so I drew one where it was)

vRealize Suite Lifecycle Manager – Resume Request

This time the step and eventually the entire request finished successfully. From the vCenter perspective the result will look like this:

vSphere Client – vRealize Log Insight cluster VMs

This process is repeated for the second cluster / environment, leaving us with two environments, each with a vRealize Log Insight cluster:

vRealize Suite Lifecycle Manager – Two environments with vRealize Log Insight

The next step is to set up message forwarding, so that the “inner” cluster will receive also the messages from the devices logging to the “outer” cluster, with only allowing SSL secured traffic from that cluster to the other on the firewall between the clusters.
Before configuring the two vRLI clusters we first need to export the certificate for the “inner” cluster, which was created separately using the vRSLCM:
(If the same certificate is used for both environments, e.g. subject alternative name=*.”parent.domain”, you can skip this)

vRealize Suite Lifecycle Manager – Settings / Certificate

The certificate is imported into all (four) nodes of the forwarding cluster (“outer”) sequentially like shown below or described in the official documentation, followed by a reboot:

SSH to vRealize Log Insight cluster VM

The receiving (“inner”) cluster can be configured to accept only SSL encrypted traffic: (optionally)

vRealize Log Insight – SSL Configuration

Finally the FQDN for the virtual IP of the the “inner” cluster is added as event forwarding destination in the configuration page of the “outer” cluster. The protocol drop-down should be left on “Ingestion API” as changing to “Syslog” will overwrite the original source IPs of the logging entries. After checking the “Use SSL” box verify the connection by using the “Test” button:

vRealize Log Insight – Event Forwarding

If no filters are added here all events received by that vRLI cluster will also be available on the other one.

For testing the setup I configured a NSX-T manager, placed at the “inner” management cluster, to log directly onto the “inner” cluster and a couple of edge VMs, which were deployed to the “outer” edge cluster, as described here.

Deploying vROPS 7.0 with vRealize LCM 2.0

In my previous post I described how to deploy the vRealize Lifecycle Manager 2.0 and import product binaries and patches.
Now it is time to make use of it to deploy the first vRealize product: vRealize Operations Manager.
There are some more steps, which you need to complete first, like generating a certificate or certificate signing request, and also some optional tasks, like adding an identity manager or Active Directory association. As they are described quite well in the official documentation I will skip those here.

Before you can add an environment (the term used for deploying vRealize products) a vCenter has to be added. The documentation states how to add a user with only the necessary roles, but for testing purposes you can also use the default administrator SSO account.

Add a Data Center to vRealize Suite Lifecycle Manager

If you have an isolated environment the request to add a vCenter will look like the above screenshot, as it can’t get patches from the internet, but it will still work.
In the “Create Environment” screen you can select which products you want to deploy. For each product you need to select the version and the deployment type:

vRealize Suite Lifecycle Manager – Create Environment screen

Next to the deployment type each product has a small “info” icon. Upon clicking that the details to each type are displayed:

vRealize Suite Lifecycle Manager – Create Environment screen (vROPS deployment types)

After selecting your desired products you have to accept the license agreements and fill in details like license keys, deployment options, IP addresses, host names etc.

vRealize Suite Lifecycle Manager – Create Environment screen (EULA & deployment parameters)

After putting in all necessary information a pre-check is performed:

vRealize Suite Lifecycle Manager – Create Environment screen (Pre-check)

The pre-check verifies the availability of your DNS servers, datastores and so on:

vRealize Suite Lifecycle Manager – Create Environment screen (Pre-check tasks)

After submitting the LCM creates the environment according to your input:

vRealize Suite Lifecycle Manager – Create Environment screen (Submitted)

As I made a mistake in the DNS server configuration the request failed.

vRealize Suite Lifecycle Manager – Create Environment screen (Failed)

Upon clicking “View Request Details” a more detailed view is presented. (see screenshot below)
Before deleting the environment and giving it another shot after having the mistake fixed you should export the configuration. Two options are offered: Simple or Advanced. I picked simple, which lets you download most of the parameters you entered as a JSON file.

vRealize Suite Lifecycle Manager – Create Environment screen (Failed, details)

The red info icon in the lower left corner gives even more details. In my case the successfully deployed master node was not reachable because of the DNS misconfiguration mentioned above.

In the “Create Environment” screen you can paste the contents of the saved JSON file (see above) to speed up the process. This brings you directly to the pre-check step. However you still need to go back one step and select your NTP servers – this doesn’t seem to be included in the JSON configuration.
While the environment creation request is in progress you can also see details:

vRealize Suite Lifecycle Manager – Create Environment screen (In progress, details)

Finally the request finished successfully. Some steps were left out, probably because this is a single node deployment and not a “real” cluster…

vRealize Suite Lifecycle Manager – Create Environment screen (Finished, details)

After the environment is created you can (and should) enable health checks via the menu which open when you click the three dots in the upper right corner of the request box. This menu also offers you to download logs and export the configuration, as done before.

vRealize Suite Lifecycle Manager – Create Environment screen (Enable health checks)

The first task I am going to do with the newly deployed vROPS is to install the HF3 security fix imported earlier:

vRealize Suite Lifecycle Manager – Environment details screen

Just select the patch, click “Next” to review and install:

vRealize Suite Lifecycle Manager – Environment details screen (Install Patch)

You can monitor the patch installation progress:

vRealize Suite Lifecycle Manager – Environment details screen (Installing patch in progress)

To be able to use the integrated Content Management you have to configure the environment as an endpoint. Just click the link “Edit” which appears when clicking on the three dots next to the list element:

vRealize Suite Lifecycle Manager – Content management: Endpoints screen

First confirm or modify the credentials entered earlier and test the connection:

vRealize Suite Lifecycle Manager – Content management: Edit Content Endpoint

Finally you have four checkboxes to selecht your desired Policy Settings:

vRealize Suite Lifecycle Manager – Content management: Edit Content Endpoint (Policy Settings)

I will pick up the Content Management section in another blog post.
Up until then the vROPS deployed using the vRealize Suite LCM can be used as usual by opening the web GUI. It asks you to set your currency (can’t be modified later on!) and is ready to fill its dashboards with data as soon as you configure the parameters and credentials for the solutions you want to monitor, e.g. vCenter:

vRealize Operations Manager – Configure Solutions
vRealize Operations Manager – Configure currency
vRealize Operations Manager – Configure currency (e.g. set to EUR)

Deploying and patching vRealize Suite Lifecycle Manager 2.0

Another customer, another project – again the need to deploy a couple of vRealize components (Log Insight, Network Insight, Operations Manager, Automation & more).
Why not use the same helper tool the VMware Cloud Foundation uses to deploy “vROPS” and “vRA”?

VMware describes this management appliance as follows:

vRealize Suite Lifecycle Manager automates install, configuration, upgrade, patch, configuration management, drift remediation and health from within a single pane of glass, thereby freeing IT Managers/Cloud admin resources to focus on business-critical initiatives, while improving time to value (TTV), reliability and consistency. Automates Day 0 to Day 2 operations of the entire vRealize Suite, enabling simplified operational experience for customers.

https://blogs.vmware.com/management/2018/09/vrealize-suite-lifecycle-manager-2-0-whats-new.html

Download and deployment of the appliance’s OVA file is pretty straight forward as with most of VMware’s current products. After starting the newly created VM in the vCenter client you can log in with the default credentials “admin@localhost” / “vmware”, as described in the documentation.

Some patches are available and can be downloaded from my.vmware.com and applied to the VM via the web GUI pretty easily.

Patches available in December 2018 for VMware vRealize Suite Lifecycle Manager 2.0

For being able to use the current versions of “vRA” and “vRLI” you also need to install a product support pack available on the VMware marketplace. For downloading you need to click the “Try” button on the right hand side. The screenshot on there shows how to install the “.pspak” file.
After the pack is applied the product versions shown in the following screenshots are supported:

vRealize Product versions supported by vRLCM 2.0.0.2

The vRealize Suite LCM first needs to import the binaries of the products which are supposed to be deployed. If you are at a site with internet access you can use the integrated “My VMware downloads” option.
At an isolated site however the easiest way for me was to upload the required OVA files into the LCM VM, e.g. with WinSCP. After connecting with the “root” user (needs to set a password first) change into the “/data” folder and create a new directory (e.g. called “binary_import”) and copy everything into there.
Afterwards import the binaries from the web GUI as described in the documentation (local location type, base location = “/data/binary_import”, discover, add).
When the LCM is finished with discovering and mapping the product binaries and importing the patches the GUI should look like this:

Succesfully mapped most recent product binaries of vROPS, vRLI & vRNI supported by vRLCM 2.0.0.2 (above) and ciritical product patches (below)

After the holiday break the next steps will be to deploy and manage the vRealize Suite components needed…