iXsystems TrueNAS CORE and VMware

Overview

iXsystems publishes a White Paper, titled TrueNAS for VMware, which provides Best Practices for VMware. In this post, I am going to breakdown each of the best practices with an example. Hopefully this will help others, newcomers and experts, alike, at an optimally configured TrueNAS. For me, it will help ease the pain of research in the future for each of these concepts.

Caveats

A couple things to note.

1. I do not use NFS for storage with VMware. I will always prefer block storage, so iSCSI is my choice of storage protocol.
2. I have TrueNAS CORE 13 installed on hardware. I will use a virtual machine for a lot of the demo purposes for this post and will provide virtual machine specific notes where applicable.

Use TrueNAS version 11.3 or later for iSCSI or NFS.

Like I mentioned in the Caveats section above, I use iSCSI as I prefer block storage over file storage. As of this writing, the current version of TrueNAS CORE is 13.0-U6.1. So as long as you grab a new copy, this should already be met.

---

Provision the datastore using iSCSI (block storage) using device-based extents (zvols).

For this recommendation, we first need to configure a Storage Pool. There is actually another recommendation about vdevs a little further down in the recommendations that we will discuss next, since it is related.

For best performance, configure the disks into at least 5 mirrored vdevs.

For this to take place, you will need a minimum of 10 disks. If you are new to TrueNAS and wondering about the difference between disks, pools, vdevs, and zvols, let me try to explain these in plain English as I understand them.

Disks – disks are the basic building block of storage. These are the physical devices that you plug into the server if you are using hardware. If you are provisioning disks on a TrueNAS CORE virtual machine, you will need to add the following Advanced Parameter to the virtual machine to avoid a warning regarding non-unique serial numbers.

disk.enableUUID = "TRUE"

Pools – pools are an aggregate of disks. This means that you can build a pool of storage with a collection of one or more disks. From this pool, you are able to carve the pool up into smaller units, but the data is redundantly stored (not protected — that’s what backups are for) in a software RAID, using ZFS. Pools can be built for a specific purpose, therefore the underlying disks also matter.

For purposes of my lab, I am only looking for general purpose storage. I do not have anything that is input/output (I/O) sensitive, such as a highly transactional database. This doesn’t mean I can’t run a database on my general purpose storage pool, it just means I most likely wouldn’t get the best I/O possible out of TrueNAS and my disks.

You must pay attention to the ZFS configuration of the pool so as not to be surprised with available raw capacity, such as in this example. If you have five 1.2 TB HDDs installed (1.091 TB actual), you can pool all five disks together for about 3.27 TiB (Tebibyte) of raw capacity with Raid-Z2. This is an example of an expensive waste of possible raw capacity. There is a ZFS Capacity Calculator on the TrueNAS website that may be helpful to use prior to purchasing disks or provisioning storage pools.

One of the recommendations listed for iSCSI depends on the importance of the data. If the data is important, it is recommended to set the Sync option to Always in the pool options.

vdev (Virtual Device) – a virtual device is a subset of a pool. TrueNAS has various types of vdevs, including data, cache, log, hot spare, metadata, and dedup.

So, a virtual device (vdev) is a pool. You use the Storage > Pools menu item to configure. Now getting back to the parent heading of this recommendation, For best performance, configure the disks into at least 5 mirrored vdevs, we would need a minimum of 10 free disks. Mirrors require at least two disks. Think about how expensive that is, not just monetarily! For 10 TiB of raw capacity, you would need to purchase disks aggregating over 20 TB! You need hardware capable of this type of disk density, most likely a Small Form-Factor (SFF) server.

The more vdevs we can create, the more concurrent read and write operations that can take place. This is extremely important advice to adhere to with virtualization. Since my disks are spinning disks (HDDs) (or spinning rust as a colleague of mine put it, recently) I do not have the benefit of randomized access like what’s experienced with solid state hard drives (SSDs). So if I pool my eight disks as a RAID-Z2 pool, I am only going to get two concurrent read/write requests. If I made four mirrored vdevs, then I could theoretically get up to four concurrent read/write requests. The physical server I am using for TrueNAS is an older server with capacity for only eight hard drives. The most mirrored vdevs I could create would be four and would be really expensive for a lab. Being that it is a lab and I am budget constrained, I have a few choices to get the best performance. I could obviously purchase a new server with greater disk density. I could max out the RAM on the server since ZFS prefers RAM. Alternatively, I could swap out all of my spinning disks (HDDs) with solid state hard drives (SSDs). The key here, though, was budget constrained. So I will have to make due with what I have and hope the 64 GB of RAM in the system will be enough.

For the example, I have a virtual machine created with thin provisioned disks. This is for demo purposes only and should not be used for a production system whatsoever as performance would likely render the environment not very usable.

Let’s create the five mirrored data vdevs to illustrate this recommendation.

Open up Pool Manager. Go to Storage > Pools and select Add.

The selection will default to Create new pool. Select the Create Pool button.

Available disks will be on the left side of the window. Provide a name for the pool and select two available disks. Click the arrow pointing to the Data VDevs side. Make sure the redundancy type is set to Mirror. When finished, select the Create button.

Select the Confirm checkbox and then Create Pool on the Warning pop-up.

Once the pool is created, select the Pool Actions (gear icon) and select Add Vdevs.

Repeat this process and keep choosing two Available Disks until you have at least five mirrored vdevs or run out of Available Disks. Select Add VDEVS after each addition.

Select the Confirm checkbox and select the Add VDEVS button after each addition.

After adding ten 1.2 TB HDDs and mirroring them, I am left with 5.3 TiB of available capacity.

zvols (ZFS Volume) – a zvol represents a block device, hence the recommendation…Provision the datastore using iSCSI (block storage) using device-based extents (zvols).

You create a zvol by navigating to the storage pool, selecting the vertical ellipsis, and selecting Add Zvol.

Provide a Zvol name, Comments, Size for this zvol, Sync option, Compression level, ZFS Deduplication option, and select the Sparse option if the zvol should be thin provisioned. (Note, thin provisioning here is not the same as thin provisioning at the VMware vSphere/ESXi side. The concept is the same, but the thin provisioning, here, applies to the storage on the Storage Area Network (SAN) side.) Select the Submit button when ready.

---

---

Create a tunable with the Variable name of vfs.zfs.l2arc_noprefetch, a Value of 1, and a Type of Sysctl

Create a System Tunable (System > Tunables)

Variable: vfs.zfs.l2arc_noprefetch
Value: 1
Type: SYSCTL
Description: Best Practices for VAAI
Enabled: yes

---

Isolate the storage traffic from other networking traffic using VLANs, network segmentation, switches dedicated for
iSCSI traffic, and jumbo frames with an MTU of 9000

Again, I am doing this in a lab, so I am not going to have access to a dedicated storage switch for only two virtualization hosts. I will rely on network segmentation using VLANs. Every switch manufacturer that supports 802.1q has a different method to enable Jumbo Frames, any Maximum Transmission Unit (MTU) greater than 1500 bytes. The important thing to remember when enabling Jumbo frames is that it must be enabled end-to-end. Physical switches to virtual switches and in-turn VMkernel adapters.

---

Do not use VMware iSCSI port binding. Instead, each VMkernel adapter used for iSCSI traffic should be on a different IP subnet that matches the TrueNAS network configuration.

I am going to combine this recommendation with the next one, explained below.

Port binding is necessary if using the same IP subnet (a flat network). Here is a document from VMware by Broadcom on Considerations for using software iSCSI port binding in ESXi. The gist of the document is that port binding is required when on the same IP subnet. Otherwise, VMkernel adapters configured on different IP subnets do not need port binding.

Note: I actually did not realize this myself until I came across this document regarding Best Practices. For years, I have always created two VMkernel adapters, disabled one side of the link (or created separate switches if enough vmnics), and bound the ports to the iSCSI storage adapter. After reading this, I was happy to learn this and reconfigured my lab.

My iSCSI networks are always Layer 2. I never route iSCSI traffic. When configuring for Multipath Input/Output (MPIO) (discussed in the next recommendation), your TrueNAS CORE will need more than one Portal. One portal provides one path. More portals equal more paths. Since I am running on 10 Gbps physical links, I am happy to provide two paths.

On each host, a pairing VMkernel adapter must be configured for each portal. As you can see below, I have two distributed port groups configured.

VLAN 165 services the 172.16.5.0/29 network.
VLAN 166 services the 172.16.6.0/29 network.

Use the VMware Round Robin PSP (Path Selection Plug-in) for MPIO and adjust the Round Robin IOPS limit to 8.

Okay, admittedly, there a a lot going on with these two recommendations and I will do my absolute best to make it really simple.

MPIO stands for Multipath Input/Output. It allows multiple paths for storage traffic to traverse back and forth. Sorry to be verbose, but stick with me. Use the VMware Round Robin PSP (Path Selection Plug-in) requires administrators to manually choose a policy per storage device, per host. This is probably okay if you have a handful of storage devices and hosts, but can quickly get out of hand with more than that. Additionally, there is no way to set the IOPS rate in the user interface.

For this reason alone, I would recommend configuring a Storage Array Type Plug-In (SATP) Claim Rule (see next section for how to configure this). Since the only way to configure the IOPS (Input/Output Operations per Second) is on the command line, anyways, it makes sense to configure this with the SATP Claim Rule.

---

Storage Array Type Plug-In (SATP) Claim Rule for TrueNAS Core in VMware

esxcli storage nmp device list

The following bold lines are needed for future commands.

esxcli storage nmp device list
naa.6589cfc000000bd3ada8f2c2b700b820
   Device Display Name: TrueNAS iSCSI Disk (naa.6589cfc000000bd3ada8f2c2b700b820)
   Storage Array Type: VMW_SATP_ALUA
   Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}
   Path Selection Policy: VMW_PSP_RR
   Path Selection Policy Device Config: {policy=rr,iops=1000,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
   Path Selection Policy Device Custom Config:
   Working Paths: vmhba64:C0:T1:L1, vmhba64:C1:T1:L1
   Is USB: false

esxcli storage nmp psp roundrobin deviceconfig set

This command is useful for testing or temporarily setting options, but is not persistent. It will not survive a host reboot. The –iops=8 is set due to the recommendation of iXsystems. A lot of vendors use –iops=1. This was not an oversight or a mistake, however. This comes from the reference I provided. If you search the web, you will see plenty of posts where people have set this to –iops=1. I am sure it works for them, but I am sticking with the recommendation for now until I have reason to change it.

esxcli storage nmp psp roundrobin deviceconfig set --device=naa.6589cfc000000bd3ada8f2c2b700b820 --iops=8 --type=iops

esxcli storage core device list

This command provides more information for us to configure the SATP Claim Rule. If you have a lot of devices, you can filter the list by providing a single device with the -d switch. I will bold the information that is needed from the output.

esxcli storage core device list -d naa.6589cfc000000bd3ada8f2c2b700b820
naa.6589cfc000000bd3ada8f2c2b700b820
   Display Name: TrueNAS iSCSI Disk (naa.6589cfc000000bd3ada8f2c2b700b820)
   Has Settable Display Name: true
   Size: 524288
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.6589cfc000000bd3ada8f2c2b700b820
   Vendor: TrueNAS
   Model: iSCSI Disk
   Revision: 0123
   SCSI Level: 7
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is VVOL PE: false
   Is Offline: false
   Is Perennially Reserved: false
   Queue Full Sample Size: 0
   Queue Full Threshold: 0
   Thin Provisioning Status: yes
   Attached Filters:
   VAAI Status: supported
   Other UIDs: vml.010001000036633230353662653033343830303100695343534920
   Is Shared Clusterwide: true
   Is SAS: false
   Is USB: false
   Is Boot Device: false
   Device Max Queue Depth: 128
   No of outstanding IOs with competing worlds: 32
   Drive Type: unknown
   RAID Level: unknown
   Number of Physical Drives: unknown
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false

esxcli storage nmp satp rule add

The following command adds the SATP claim rule using information found from the previous commands.

esxcli storage nmp satp rule add --satp=VMW_SATP_ALUA --vendor=TrueNAS --model="iSCSI Disk" --psp=VMW_PSP_RR --psp-option=iops=8 --claim-option=tpgs_on --description="TrueNAS iSCSI Claim Rule"

esxcli storage nmp satp rule list

This command will list out all the known rules in the system. There are a lot by default so I find it helpful to grep out what I am looking for. In this case, I want to verify the rule we just tried to add did in fact add and is correct.

esxcli storage nmp satp rule list | grep -i truenas

esxcli storage nmp satp rule remove

If, for whatever reason, you find you no longer need the SATP Claim Rule or found a mistake, you will have to remove the entry. You cannot modify a rule. The remove rule was a little confusing for me based on the error messages. It turns out you need to provide it with every field that has something in it to match against.

esxcli storage nmp satp rule remove --satp=VMW_SATP_ALUA --vendor=TrueNAS --model="iSCSI Disk" --psp=VMW_PSP_RR --psp-option=iops=8 --claim-option=tpgs_on --description="TrueNAS iSCSI Claim Rule"

Once you verify the SATP Claim Rule has been added and is correct, place the host into Maintenance Mode and reboot it for the rule to take effect.

---

Configure VMs using VMware paravirtual SCSI controllers and VMXNET3 network adapters.

This is combined with the next section.

---

Install the VMware client tools.

I am including these two recommendations for completeness sake.

Every virtual machine should be configured with a VMware Paravirtual SCSI controller and a VMXNET 3 network adapter. This goes to say that VMware Tools is also necessary to take advantage of these technologies.

---

Conclusion

This was a long post, but hopefully some of the explanations will help squeeze more I/O out of your TrueNAS CORE. I learned a lot while researching all of this and I hope I captured it in a way that can help beginners as well as seasoned TrueNAS CORE administrators. I always welcome constructive criticism, please let me know if anything needs to be reinforced or clarified.