VMware ESXi 6.5 U1 and VSAN 6.6.1 Lab Upgrade

VMware vSphere 6.5 U1 has been released last week, Friday, so I had time to upgrade the lab over the weekend. Mixed results as you'll see later in this post, but after the upgrade, I'm running the latest version – the vSphere 6.5 U1. The lab runs a Supermicro Motherboard which has a certified storage controller LSI 3008, including the vSAN 6.6.

Update: The lab is still certified with the LSI 3008 and stays “green” within the vSAN cluster. You can check it through by selecting the vSAN cluster via vSphere web client > Monitor > vSAN > Health.

So, this post is a recap of the lab upgrade to the latest version of VMware vSphere 6.5 U1. I have installed VMware vCenter Server Appliance (VCSA) since 6.5, so the upgrade to the VCSA 6.5 U1 was easy. Download the upgrade ISO file from VMware Patch portal > Attach the ISO to the VM > Do the upgrade through the VAMI user interface > Reboot > Done in 15 minutes.

The upgrade of the individual hosts was a different story. Kind of. One host failed to upgrade and had a failure of a USB stick. So I had to reinstall the host by using a new USB stick. This was fine. Note that this host was a part of a 2-node vSAN cluster and it stored half of the vSAN components.

After reinstalling the host, configuring the network, VMKernel adapters for vSAN traffic, the host has been added to the vSAN cluster and automatically triggered the vSAN resync. The single disk group which has been untouched has been found by vSAN. The system automatically triggered the vSAN resync which took almost 1 h to finish.

The other two hosts were updated by using VMware Update Manager (VUM). Since the vSphere 6.5 and VCSA 6.5, the VUM is pre-installed within the appliance. So the steps are very easy.

Import the full ESXi 6.5 U1 iso into the VUM > Create Baseline > Select host > Attach baseline > Remediate > Done.

Basically, the same way as I proceed with the 6.0 to 6.5 upgrade via VUM in my post a while back – How to upgrade an ESXi 6.0 to ESXi 6.5 via VMware Update Manager.

While VMware VSAN 6.6.1 did not need any further disk upgrades (still the disk format v5), the system was showing few yellow checks. ~~And it is because the LSI 3008 HBA has fallen off the VSAN Hardware Compatibility List (~~HCL~~) for that release. The only way to stay “all green” would be to silence the hardware~~ HCL ~~checks through Ruby vSphere Console (RVC), but I prefer to leave the warnings.~~

There are possible firmware upgrades for the LSI 3008 (for users which are running labs on this system). The LSI firmware upgrade can be done either via the CLI or with a help of the vSAN UI. Note that in my case, the button which should facilitate the FW upgrade was grayed out despite the installation of vmware-esx-sas3flash.vib (tested v13 and v12 of the vib, but none of them worked).

Updated view the September 24th:

Here is the original view with the LSI marked as “Yellow”, but then apparently the LSI 3008 has made it to the HCL.

I went to the VMware HCL page to see if there is a newer driver for the 6.5 U1 release. After searching for the driver got the access to this listing page.

You can see below that you have to respect firmware release and driver version combination. I picked up the latest supported driver for FW 12. I might do the FW upgrade at a later stage, but for now, I'm just fine to update my VMware driver.

The driver update:

After downloading the driver v13, it's fairly simple. Just unzip the file. You'll find two files, as usually. One is a VIB and one is a ZIP file. Copy one of those files to a storage visible by your host.

You can use either of the commands below, for the driver update. If you're using the VIB just use the first command, if not, use the latter one.

Issue the following command (full path to the file must be specified):

esxcli software vib install -v {VIBFILE}

esxcli software vib install -d {OFFLINE_BUNDLE}

In my case I entered:

esxcli software vib install -d /vmfs/volumes/Drobo/mpt3sas-13.00.00.00-6.0-offline_bundle-3755736.zip

And we need to reboot the host…

While you can perfectly find the firmware version through the VSAN User interface, sometimes you need to use CLI. If you're wondering which commands to use, just keep reading.

Host HBA Firmware Version Check

Use this command to display your host's HBA/driver combination via CLI.

/usr/lib/vmware/vmkmgmt_keyval/vmkmgmt_keyval -d

This gives you the output of your HBA. Append this to the command (modified) as this (my example).

/usr/lib/vmware/vmkmgmt_keyval/vmkmgmt_keyval -l -i lsi_msgpt3_0/LSI

My output and detailed view:

Update: New esx cli commands shows also which controllers are used by vSAN cluster and what's the queue depth.

Just connect to any ESXi part of the vSAN cluster, via putty, and enter:

esxcli vsan debug controller list

You'll get this view…

VMware vSphere Windows C++ client

The latest version of the Windows vSphere Client (VMware vSphere Client 6.0 Update 3, VMware-viclient-all-6.0.0-5112508.exe) , available from this VMware KB page no longer works on ESXi 6.5 U1. If you're still planning to use this old Windows client, you can't upgrade to ESXi 6.5U1. (unless there is some kind of hack).

You have the option to use the HTML 5 vSphere Client which covers perhaps 80 percent of administration needs (not vSAN). For individual ESXi hosts, you're left with the HTML 5 host client.

Wrap up:

VMware VSAN is strict on hardware. We know that. The lab hardware is lab hardware. It does not really matter to me whether i is certified or not, whether it shows me green checks or not. Of course, it would be better, but all I basically need is the possibility to use it. I was expecting to happen sooner or later. The lab runs vSAN since 2015. If it was a production system, this would be different. But hey, it's a lab. I'm only using it for learning and testing.

But one thing is sure and it is the fact that you must absolutely respect the Firmware/Driver combination for your HBA and NICs if you want your VSAN dashboard to be full of green checks. And it's perfectly sure that you must check for that before you start the upgrade process.

Before jump into the upgrade wagon, make sure that you check if you upgrade to vSphere 6.5 U1 you still be supported. Make sure that all your external monitoring, replication products are supported on vSphere 6.5 U1 too. It seems that the vSphere 6.5 U1 is kind of more polished with fewer issues, but it is only my feeling.