Table of Contents
Overview
The networking component for the nested VCF lab is very important. There are certain things that must be configured for a successful deployment. These settings do not apply in a production scenario where the physical hosts are connected to Top-of-Rack Switches. Doing so could open up a serious security situation.
On the physical host.
- The Virtual Switch (vSwitch0) should have the Maximum Transmission Unit (MTU) set to 9000 (also known as Jumbo Frames)
- The Security settings (Promiscuous mode, MAC address changes, and Forged transmits) on vSwitch0 should all be set to Accept
UPDATE: June 2024 – vSphere Distributed Switch (vDS)
Originally, I had a single host, so I used a vSphere Standard Switch (VSS). I ran out of compute with all the things VCF so I added a second host. Now, I am using a vSphere Distributed Switch. There were a few things I needed to set to get this working correctly. It took weeks of troubleshooting and finally a colleague had to assist me. The resolution is not intuitive and in fact, is very frustrating.
Build out the vSphere Distributed Switch and add it to your hosts. Ensure the MTU is set to 9000, the Distributed Port Groups will inherit this setting. The only difference here is that the Security Policy is set at the Distributed Port Group level, not the switch level.
I built out a Distributed Port Group called vds-Trunk. Navigate to the Configure tab, select Policies under Settings.
Set the Security policy like you would on the Standard Switch.
- Promiscuous mode: Accept
- MAC address changes: Accept
- Forged transmits: Accept
Ensure the MAC Learning is Disabled! This is important and not intuitive at all! I had this set, originally, and had all kinds of trouble when the nested ESXi guests were migrated to the other host. Also, other VMkernel adapter addresses were not being learned by the underlying physical switch.
One thing my colleague did offer was to make sure to delete the original vmk0 VMkernel adapter, which assumes the physical address (the MAC address) assigned. So you can delete and recreate the Management VMkernel adapter using the virtual machine console.
Nested Firewall
I use a Firewalla Gold for a home firewall. This is a great firewall…for home use. I use this primarily for home, but I also want to be able to have a home lab that closely resembles an Enterprise network. Unfortunately, this device does not have a way to easily adjust the MTU or have native Border Gateway Protocol (BGP) support. For this reason, I end up installing a pfSense firewall on my physical ESXi host.
For this to work, I add two VMXNET3 Network Adapters. One is on my home lab network, VLAN 92, and the other adapter is configured with a VLAN ID of 4095 and labeled All-VLAN IDs. In VMware, VLAN 4095 enables Virtual Guest Tagging (VGT) and means the virtual machine handles the VLAN traffic.
Lab Physical Diagram
pfSense to Physical ESXi Logical Diagram
Create the pfSense Interfaces
Take the Deployment Parameter workbook and create the interfaces in pfSense. Make sure to adjust the MTU on the LAN interface to 9000 before adjusting the MTU of the sub-interfaces.
Update: The deployment parameters Management Domain Network in VCF 5.1.0 broke the management networks up. Now there is a VM Management Network which is used for all the management guests, such as SDDC Manager, vCenter Server Appliance, and the NSX Managers. Then there is a Management Network which is intended to be used for the ESXi host kernel interfaces (vmk0).
Create a DHCP Scope for the NSX-T Host Overlay Network
DHCP is recommended for the NSX-T Host Overlay network. I use /24 networks, and since there are only four nested hosts, I just open the range up from 100 to 199.
Permit Traffic to the Management Gateway
When I ran the VCF configuration validation, I kept receiving a warning that the gateway was not contactable. I logged into the console of the Cloud Builder appliance and verified that I actually could. So I checked the firewall logs and saw the traffic being blocked. This was also confirmed in the VCF bring-up log.
The VCF bring-up log can be found at:
/var/log/vmware/vcf/bringup/vcf-bringup.log
I thought this was quite odd behavior. What administrator opens up these ports or has anything listening on the router interface at these addresses? Huh, interesting…
Anyways, I added an alias in pfSense and allowed those ports on the appropriate interfaces. The next section will provide more detail on the pfSense.
pfSense Configuration
I am using a nested pfSense firewall in my lab due to the shortcomings of my edge firewall, unfortunately. The pains of home labs, I suppose. Either way, pfSense is quite impressive in this role and does a great job without consuming too many resources. Probably the most frustrating aspect of this set up are the firewall rules. This is not a pfSense issue. This is a security detail. Sure, I could open my firewall wide open, but who in their security conscience mind would do such a thing?
Note: The following has been updated for VMware VCF 5.1.0 and may not be appropriate for your version.
Firewall Aliases
Aliases in pfSense are a way to provide a friendly name to a group of either IPs, Ports, or URLs. As you can see in the image below, instead of using IP addresses for my VMware ESXi hosts, I added an alias with the name, ESXiHosts, and provided all the ESXi hosts in my lab. This will save me on my rulesets as well. Now I can reference one line instead of many for the same task. Additionally, referencing by name rather than IP address is easier to read…and in the off chance IP addresses have to be altered in the future, the rulesets will still be accurate and will not need to be altered.
Firewall Rules
The first group is Floating. This has special meaning and are rarely used. I wanted this particular rule, to be able to access TCP/443 (HTTPS) across all the interfaces. One of the differences of the Floating group is that it is bi-directional. Normally, the direction is ingress to an interface.
WAN
Management Network
vMotion Network
vSAN Network
NSX Host Overlay
VM Management Network
Leave a Reply