As you know my lab got an addition this year, the Supermicro's Single CPU board a X10SRH-CLN4F model. In this post, we will be upgrading LSI 3008 HBA on the X10SRH-CLN4F. It is a board with four 1-Gigs Intel NICs, IPMI, and lots of other features which I previously missed on consumer based boards. It's certainly the memory limit which I needed to blow.
Also, I wanted to go as energy efficient as possible but still to have something “future proof” with a good performance as well. The board supports up to 512 Gb RAM, but DDR4 is still very expensive so there is 64Gb of RAM for the moment. You might want to check the series from the start – Efficient Home Server – Start with an Efficient Power Supply.
Update 3: I just found out that my LSI 3008 controller which is used by VSAN is still “Green” via VMware HCL health checks. This controller is a SAS/SATA controller, 12GBps which can drive up to 8 disks. In my case, I use it for vSAN and right after the upgrade, the controller seems to fall off the HCL, but a few months later, it's back again.
Note that while this controller can do software raid, I'm not using it. I'm passinng through the disks into my vSAN so it' is vSAN cluster which has a full management of the devices. You might be interested in reading this post about the latest lab upgrade to ESXi 6.5 and vSAN 6.6.
It seems that while VMware does not have priority on this Supermicro Motherboard (which is understandable), the controller itself is still pretty valuable.
I also tried to use the new vSAN CLI command:
esxcli vsan debug controller list
Which basically gives you not only the details about the controller, driver and firmware but also the queue depth… -:) Pretty cool. This is the way VMware is going forward. Latest news from VMworld 2017 shows that Ruby vSphere Console will be phased out in the future and VMware will pack more CLI commands and PowerCLI commandlets in order to interract with vSAN clusters.
Updated view the September 24th:
The board's built-in LSI Logic Fusion-MPT 12GSAS SAS3008 PCI-Express which has been recently added on VMware HCL for ESXi 6.0 has queue depth 600 (same as my other Dell Perc H310 flashed with IT firmware…). The difference is that the LSI 3008 is 12 GbpS SAS where the H310 is 6Gbps only… You can check yours too – how to check queue depth of your storage adapter.
This is the first time that i'll be flashing a board from UEFI mode and since I had to seek hard for the solutions I thought that spreading the knowledge is a good way to do. Now you might say, why would you bother to flash via UEFI if it can be done via bootable DOS USB stick? It can't. I tried several times and each time it throws an error….
If you receive a “Failed to initialize PAL” message while trying to flash an LSI controller card, it is most likely due to your motherboard having limited Option ROM. And the advice I found of few sites was “You will need to find another motherboard to use to flash the controller card”. Since it wasn't possible as the LSI 3008 is directly on the board…
The UEFI was something new and only to find out how to change to the USB stick within the UI it wasn't easy as the USB drives starts with fsx… like fs0, fs1 etc… so to change to the USB stick from within the UEFI you just do for example:
fs0:
And then simply dir to the directory where you have copied your firmware upgrade files.
Upgrading LSI 3008 HBA on the X10SRH-CLN4F – the necessary steps:
1. Format a USB stick and copy there the firmware upgrade files. I have downloaded the files from Supermicro's ftp site:
2. Boot your system by hitting CTRL-C to enter the LSI Bios. Once there write down the last 9 digits of the card's
3. Reboot and this time hit F11 to choose the UEFI: Built-in EFI Shell
4. You'll get to the (famous) EFI shell where you need to change to the USB and run a single command to upgrade the firmware… Yes single command, so it's actually more simple than the upgrade I've done to my Dell Perc H310 via DOS.
The EFI shell on the Supermicro board looks like this
5. Then according to the documentation, just run SMC3008T.nsh and follow the instruction to enter the last 9 digits/characters of the SAS address.
I have also recorded a video which details the whole process. So this time you'll see it going live… -:)
Watch in HD and Full Screen…
There are 8 RAM slots on the motherboard which allows boosting the single server up to a total of 512 GB of RAM when using DDR4 LRDIMMs. Or when used with lower capacity DDR4 1.2V-2133 RAM ECC sticks (8 or 16Gb RAM sticks) you can go up to 64Gb or 128Gb of RAM respectively.
In the same category at Supermicro, you can find a board less expensive, with only 2 Intel 1Gb NICs, or a board without the storage controller from LSI. Here are the 3 Single CPU boards for 2011-3 architecture with DDR4 8 memory slots, (all boards are ATX format):
- Supermicro X10SRH-CLN4F-O (with LSI 3008 SAS, SATA, IPMi) – the board I use
- Supermicro X10SRH-CF-O (with LSI 3008 SAS, SATA, 2 NICs, IPMi)
- Supermicro X10SRI-F-O ( with 2 NICs, IPMi)
The whole journey:
The 3 lab hosts I have now in the lab (https://www.vladan.fr/lab) are all based on X10SRH-CLN4F board which can go up to 512Gb RAM…
- Efficient Home Server – Start with an Efficient Power Supply
- ESXi Home lab – Get a quality and comfortable case
- Supermicro Single CPU Board for ESXi Home lab – X10SRH-CLN4F
- Supermicro Single CPU Board for ESXi Home lab – X10SRH-CLN4F – Part 2
- Supermicro Single CPU Board for ESXi Home lab – Upgrading LSI 3008 HBA on the X10SRH-CLN4F – Part 3 – this post
- Building Energy Efficient ESXi Home Lab – parts list
- Homelab – Airflow Solar System
Wrap up:
I have learned a new way to patch via UEFI. In fact, it's same (or easier) than through DOS-based bootable USB. The IT firmware can be reverted back to IR firmware as in the ZIP package there are both versions there. So in case you need a server with hardware RAID, you can use the IR version. I was actually wondering what it means the IT and IR and here is what I have found at LSI (Avago) website:
“IT” firmware maximizes the connectivity and performance aspects of the HBA. “IR” firmware offers RAID functionality via RAID 0, 1, and 10 capabilities.
Exactly what I thought.
More from ESX Virtualization:
- Free Tools
- Dedicated vSphere 6.5 Page
- What’s New in vSphere 6.5 – New Training Course from PluralSight
- How-to unlock the VMware VCSA root password?
- What is The Difference between VMware vSphere, ESXi and vCenter
- What is VMware Platform Service Controller (PSC)?
- VMware Mirage Server Infrastructure [Guide]
If you have found the post useful, just share it through your usual media channels and don't forget to subscribe to our RSS Feed, Twitter, and YouTube Channel!
Hi Vladan,
What is your expert opinion on the following SuperServer 5028D-TN4T
I’m looking to get 3 of these to roll out my 2015 lab.
Let me know your thoughts.
Best Regards,
Mo’
My lab (if we can still cal it a lab) has been created with few things in mind:
– Expandability
– low power
– performance
The Supermicro is tiny factor model where space is limited. (that’s always problem with micro servers). Especially for storage. 128 Mb of RAM might seems OK for now, but will you be sure about it in 3 years time?
However It might be a good fit for my management cluster so I can recover back my 64Gb Whitebox …. -:)
Excellent suggestion. I will get one of these for mgmt cluster and get three similar to your lab. My work however is mostly for testing production designs related to nsx. So this should be good for 2-3 years down the road.
Hi Vladan,
I am using the same board and want to use the LSI 3008 as a SATA raid. When I install ESXi 6.0, the raid 10 (or raif 5) volume is not visible in VMWare, only the hard drives are visible. Is this because I am using SATA raid?
Thank you for your reply.
John
No. The raid, you must configure it before ESXi boot. Press CTRL-c during the boot process to enter the config utility. You can use SATA drives…..
Hello Vladan,
Thank you for the answer. I have tested a few things and I think I know the reason. I have connected the sata disks to the normal sata connectors of the motherboard. I configured the RAID with the Intel bios.
Now I have ordered a SAS to 4 sata cable, and I hope and expect that the LSI chip recognize the sata drives. It is my unfamiliarity with this types of motherboards.
John
Hello Vladan,
nice description on building the environment. One question to you, how many devices did you have connected to the LSI controller ? Is it possible to add more than 8 devices ? And if yes how to do this in a desktop case ?
many thanks
Nice that you want to build an energy efficient server. But I don’t see any figures of watt useage in idle and max cpu?
True, I haven’t done that just yet….
hope to see some figures soon!
It took me a minute to find the download, so I thought I’d leave it in your comments. If this works, I’ll name my first child after you. I have an older Adaptec 6805 in my server that has been a nightmare in itself because I did a “brownfield” upgrade from LGA 1366\Server 2008 R2 based to a LGA 2011v3\ESXi. Have been getting like 30,000 ms latency when trying to read\write from anything attached to the LSI adapter with like 10Mb/s throughput.
Here’s the download:
http://ftp.supermicro.com/Driver/SAS/LSI/3008/Firmware/
Firmware update couldn’t have been easier, so good news is there will be a little Vladan running around some day. Bad news, still getting really poor read/write rates, but latency no longer seems to be an issue. Max throughput is like 30MB/s on 100K IOP drives. I know I get > 200MB/s on the same drives in a router/DC based on the SUPERMICRO MBD-A1SRi-2758F-O using the onboard SATA controller, but I’ve never really tested those drives; and I’ve seen > 400MB/s in my workstation. Any ideas? SM hasn’t really been any help on this one. Official position is although the 3008 is supported in ESXi 6, the motherboard is not, and the Samsung drives aren’t on their tested/compatible hardware list.
Question – do you know if ESXi will recognize a SATADOM? I’d like to use the SATADOM for the ESXI installation/boot device.
I’ll probably go with the X10SRT-F-O (no LSI 3008) since I need a 12Gb/s SAS with 512Mb cache and battery-backup plus at LEAST 8 drives – RAID 5 – one LUN. The drives don’t need to be high capacity – I just need a bunch of them (weird I know). So hopefully a bunch of SSDs!
BTW – love the R5 case :o)
I personally haven’t tested SATADOM just yet. But it may be supported. I invite you to check VMware communities where there is always someone ahead of time -:).
hey,
yes it will work. i use it with the same board.
Hi,
I am planing to build my home lab and i am using Intel Xeon E5-2630v4 processor. i have small question about mother board. Does every supermicro board support ESXi 6.5 ? Can i use Supermicro X10SRL-F Intel C612 So.2011-3 Quad Channel DDR4 board for home lab ?
regards
Surya
Well, the board you plan to use supports (officially) VMware vSphere up to 5.5U2. This does not mean that higher versions are not working, but I think that they’ll need an integration of newer drivers for some components such as network or storage controller. Check my ESXi lab page
Sorry Vlad a noob req I cannot passthrough my SATA HDD to my XPEnolgy VM in ESXI 6.5
If I connect them to LSI 3008 flashed in IT mode may I see them?
Really thanks
Hello Vladan,
did you tested The RAID configuration on Brodcom SAS3008 and providing this raid array to vmware?
Hi Jakub,
No because my SAS3008 is in IT mode (not RAID mode), which is necessary to use for VMware vSAN.
I have a X11SSL-CF. Trying to update the onboard LSI 3008, the firmware was deleted. I’m trying to reflash the firmware so the LSI can be used.
When you say “Format the USB” are you meaning to make it a bootable USB?
What programs are you using to enable it to be booted under UEFI, and what are the exact steps you are using to do so?
No. Not necessary, because the system boots into BIOS of the LSI 3008, in order to do the update.
Hi Vladan!
I have a cluster of SM with LSI LSI Logic Fusion-MPT 12GSAS SAS3008 PCI-Express cards. I have not found a way to figure out the FW on these cards. Just the Driver version.
I want to upgrade the ESXi 6.0 to ESXi 6.7 on the hosts, and vSAN already says that existing driver is not on the HCL.
I want to upgrade the ESXi driver on these from 06.255.12.00-8vmw.600.1.17.3029758
to 16.00.00.00-1OEM (lsi-msgpt3-16.00.00.00-1OEM.600.0.0.2768847.x86_64.vib)
and the latter says the minimum FW version is 15.00.00. How can find out what the FW version of these cards is at the moment?
Thanks in advance,
-Vitaly
I’m no longer using ESXi and VSAN on that hardware (went nested lab with VMware Workstation), so I cannot really help here. Try Google it or VMware support. Best luck.