Bits, Bytes, & Radio Waves

A Journey Through Discovery

Deploying NSX Edge Nodes

Overview

NSX Edge nodes have mystified me and have been quite a challenge figuring out. I have to admit, the number of components, getting the correct VLANs and segments configured and all in the right place has been challenging to learn for some reason. I think, after reading the current iteration of documentation and having conversations with colleagues, I finally have this sorted out!

This post assumes NSX is configured on vSphere.
vSphere Version: 8.0.3
NSX Version: 4.2.1.1

Physical Networking

ESXi Hosts

A conversation about virtual networking doesn’t make sense without writing about the physical networking.

For my lab, I have two identical Dell PowerEdge servers with two 10 Gbit/s fiber adapters uniformly configured.

The hosts are both attached to a Distributed Switch with both adapters. vmnic0 and vmnic1 are 10 Gbit/s, the unused adapters are 1 Gbit/s and not used.

Top of Rack (ToR)

I have a 10G Data Center Switch configured with VLANs and BGP.

VLANs for this post are 1610 through 1613.

Here is the BGP configuration. Notice I am simulating having two Top of Rack switches by using two separate VLAN interfaces.

The Edge Transport Node Puzzle Pieces

In no particular order, the following components need to be configured for Edge Transport Nodes to be successfully configured.

  • Distributed Port Groups on the Distributed Switch
  • Transport Node Profile
  • Hosts prepared
  • Segments
  • IP Address Pools
  • Uplink Profiles
  • Named Teaming Policy
  • Transport Zones
  • Edge Transport Nodes
  • Edge Clusters
  • Tier-0 Gateways
  • Gateway Interfaces
  • BGP ASN
  • BGP Neighbors

Order of Operations

  1. Transport Zones
  2. Compute Managers
  3. Uplink Profiles
  4. IP Address Pools
  5. Prepare uplinks (Distributed Port Groups as VLAN trunking in vSphere Client or NSX Segments in NSX)
  6. Create an NSX Edge Transport Node

If you plan to use Uniform Passthrough (UPT) mode, consult the Installation Guide for more details on requirements. I do not have SmartNIC capability in my lab to take advantage of this.

Transport Zones

System > Configuration > Fabric: Transport Zones > Transport Zones

Transport Zones control the reach of Layer 2 networks in NSX. N-VDS is a software switch that gets created on a transport node. For each transport zone that an NSX Edge belongs to, a single N-VDS gets installed on the NSX Edge.

Create two Transport Zones, an Overlay Transport Zone for internal NSX tunneling between transport nodes and a VLAN Transport Zone for uplinks external to NSX.

For Host Transport Nodes, attach both Transport Zones in the Transport Node Profile. Without the VLAN Transport Zone, NSX VLAN segments will not be accessible on the host.

For Edge Transport Nodes, attach both Transport Zones on each configured Edge Node in the N-VDS configuration.

Compute Managers

System > Configuration > Fabric: Compute Managers

Add a role and service account to vCenter and then configure a Compute Manager in NSX. This allows visibility into vSphere Resources like Distributed Switches and ESXi hosts. This is also how the Edge Transport Nodes will be deployed.

Uplink Profiles

System > Configuration > Fabric: Profiles > Uplink Profiles

An uplink is a link from the NSX Edge nodes or hypervisor nodes to the top-of-rack switches or NSX logical switches. A link is from a physical network interface on an NSX Edge node or hypervisor nodes to a switch.

Important information to take note of regarding Uplink Profiles. The Transport VLAN set in the Uplink Profile tags Overlay traffic only and is used by the Tunnel Endpoint Pools (TEP IP Pools).

Host Uplink Profile

For the Host Uplink Profile, do not configure an MTU in the Uplink Profile. It gets configured on the Distributed Switch in the vSphere Client. Since the Host TEP VLAN is configured to use VLAN 1610, set the Transport VLAN in the Uplink Profile to 1610.

Edge Transport Node Uplink Profile

For the Edge Transport Node Uplink Profile, set the Transport VLAN to 1611 for Tunnel Endpoints (TEP) and set the MTU accordingly.

Teamings

For both Uplink Profiles, I am using the [Default Teaming] policy. A Named Teaming Policy can be configured to override the default teaming policy for VLAN backed segments. Named Teaming Policies are used to steer VLAN traffic to specific uplinks.

IP Address Pools

Networking > IP Management > IP Address Pools

I created two IP Address Pools, one for Host Tunnel Endpoints (TEP) and the other for Edge Transport Node TEPs.

For Hosts, these are used in the Transport Node Profiles in the Host Switch configuration.

For Edge Transport Nodes, these are used when configuring the NSX Virtual Distributed Switch (N-VDS) information for the node.

At one time, I did not know the answer to the design decision on whether to configure the VLAN type for a Distributed Port Group as VLAN (i.e. access port) or VLAN trunking (i.e. trunk port). I was happy to finally read the following statement in the Installing NSX Edge: NSX Edge Networking Setup section of the NSX Installation Guide. That one statement makes this so much easier for me; it’s one less design decision to make!

Notice that the VLAN and tunnel port groups are configured as trunk ports. This is required.

Now, while reading the documentation, in the prerequisites, there is a statement that doesn’t make sense and is confusing.

Prepare uplinks. For example, distributed port groups as trunk in vCenter Server or NSX Segments in NSX.

  • Create distributed trunk port groups in VMware vCenter for management, TEP and overlay networks if you plan to connect NSX Edge network interfaces to a VDS in VMware vCenter.

First, do not create a VLAN trunking Distributed Port Group for Management. The Management Distributed Port Group is used when configuring the Management Interface for the Edge Transport Node, but there is no place to specify a VLAN and the Edge Transport Node will fail to register.

Instead, create a new VLAN Distributed Port Group or use a Distributed Port Group already configured for VM Management. For example, I have a VLAN 165 configured for my virtual machine management like vCenter and NSX Manager.

Second, when reading, “TEP and overlay networks”, it seems like two separate Distributed Port Groups need to be created. Two separate distributed port groups do need to be created, but only one Distributed Port Group needs to be created for the Overlay. The other Distributed Port Group that needs to be configured is for the Edge Transport Node Uplinks. Yes this is very confusing (and frustrating) and yes, this has also tripped me up tremendously!

For the Overlay network, a single VLAN trunking Distributed Port Group is appropriate.

For the Edge Transport Node Uplinks, one or two Distributed Port Groups can be configured. One Distributed Port Group is needed if only using one VLAN transport zone and two Distributed Port Groups are needed if using two VLAN transport zones for redundant uplinks. For this example, only one Distributed Port Group is configured.

Where The Networks Are Used

I have had confusion with this in the past, so I wanted to take a moment to distill this down and illustrate this with examples.

The Management network is used when configuring the Edge Transport Node. It is in the Configure Node Settings step when creating the Edge Transport Node.

The Overlay network, that is to say, the network used for the Tunnel Endpoints (TEP), is configured in the Uplink Profile for the Edge Transport Node.

This Uplink Profile is then selected in the Configure NSX step when creating the Edge Transport Node. This is what sets the Teaming Policy Uplink Mapping for the N-VDS switch.

Finally, we have the Edge Transport Node Uplinks. These can also be found in the Configure NSX step when creating the Edge Transport Node. The number of and name of these Uplinks are configured in the selected Uplink Profile.

Add Edge Node

Ok, this is where the rubber meets the road. Assuming that the above has all been configured, the Edge Transport Node creation should deploy successfully.

Provide a Name, Host name/FQDN, optional Description, and select the appropriate Form Factor.

Provide Passwords as necessary and choose whether to Allow SSH Login. For a production install, I would recommend leaving Allow SSH Login disabled. The administrator can always login at the virtual machine console.

Choose a Compute Manager, Cluster, optional Resource Pool, optional Host, and Datastore.

Select an appropriate Management IP Assignment and Type. Provide a Management IP in CIDR notation, Default Gateway, and a Management Interface that corresponds to the Management IP addressing.

Provide a list of Search Domain Names, DNS Servers, and NTP Servers.

Choose an Edge Switch Name or leave the default. Choose one Transport Zone for Overlay traffic and one or more for VLAN Transport Zones. Choose an appropriate Uplink Profile. Select the IP Address Type (TEP), IPv4 Assignment (TEP), and IPv4 Pool (if using an IP Pool). Finally, for the Teaming Policy Uplink Mapping, select the appropriate Distributed Port Group(s) if using a Distributed Switch with vSphere or the appropriate NSX VLAN Segment(s). The number of uplinks presented directly corresponds to the selected Uplink Profile.

An Edge Transport Node can belong to one overlay zone and multiple VLAN transport zones. VLAN transport zone provides the uplink access.

N-VDS Switch

  • One vNIC is dedicated to Management traffic
  • One vNIC is dedicated to Overlay traffic (fp-eth0)
  • Two vNICs are dedicated to external traffic (fp-eth1, fp-eth2)

Leave a Reply

Your email address will not be published. Required fields are marked *