It's time to do another home lab build! This time I want to break the RAM barrier for Homelab white box. In the past when I use to build a white boxes at the end I got always stuck not by CPU performance, but RAM limit. With 24Gb, 32Gb or 64Gb RAM respectively. No matter how hard you seek for a board when the RAM demands grow by running more and more VMs, the only way to go is to purchase a server motherboard, even if now with Haswell-EP you can go up to 64 Gb of RAM on desktop boards, it's still not enough for me and for majority of virtualization folks out there. I wanted to get something “future prove”, something that would still be relevant from 3 to 5 years from now on.
I also wanted to build a very Power efficient ESXi Home lab server which will not only be evolutive from the RAM perspective, but also from the storage or network perspective. If you're going after hyper converged lab architecture than this build might be a first part of the puzzle where you can add 2-3 other hosts later.
Imagine to pick up an Intel Ethernet Converged Network Adapters XL710 10/40 GbE and 10 or 40 Gb switch and you get a very efficient and performant VSAN cluster homelab. You might also want to build a All-In-One box and run everything nested, but you'll still need as much ram as possible. The Supermicro board can go up to 512 Gb of RAM. Sky WAF is the limit …
The new Xeon E5 V3 platform is very interesting as the C612 Intel chipset can handle 10 SATA drives. By picking the Supermicro X10SRH-CLN4F board the onboard LSI 3008 SW based controller can handle 8x SAS3 (12Gbps) in RAID 0, 1, 10. If you don't want to go the “SAS” way you can use SATA on the same controller. Coupled with an energy efficient Xeon E5-2630L V3 which has 55W TDP only, it's possible to build a very efficient home server. The Xeon E5 2630L V3 is a 1.8GHz 8 core CPU and has all specs like the higher frequency brothers including 20Mb cache, Intel VT-X …
The Supermicro X10SRH-CLN4F board is single CPU board based on 2011-3 socket supporting latest Intel Xeon E5-26xx V3 or E5-16xx v3 series. The board has also built-in LSI SAS 3008 controller and Quad port Intel Intel® i350 Gigabit LAN ports. No 10GbE, but 4 gigabit ports allows to get started with. (Note: The LSI 3008 is not on VMware VSAN hcl, but and it has a queue depth 600 which is fine for VSAN).
When few years back I was building one of my first Whiteboxes, the built-in LAN ports are sometimes not recognized, like the Realtek 8169 Gb LAN NIC. But then again you must use some dual or quad add-on NICs to have at least 3 NICs in each box…
8 DIMM slots – single CPU Supermicro X10SRH-CLN4F board with built-in LSI 3008 and Intel i350-AM4 Quad Port Gigabit Ethernet. 8 SATA ports 6Gbps and 8 SAS ports 12Gbps.
There are 8 RAM slots which allow boosting the single server up to a total of 512 GB of RAM when using DDR4 LRDIMMs. Or when used with lower capacity DDR4 1.2V-2133 RAM ECC sticks (8 or 16Gb RAM sticks) you can go up to 64Gb or 128Gb of RAM respectively.
The board has an IPMI for remote management and Supermicro is one of the manufacturers which do not require an additional license to use this feature (Fujitsu, HP, Dell, do you hear me?). This is one of the reasons that Supermicro boards are so popular and that's why I decided to take the plunge and start to build a ESXi platform based on this board.
The X10SRH-CLN4F Specs:
- Single socket R3 (LGA 2011) supports Intel® Xeon® processor E5-2600 v3 and E5-1600 v3 family
- Intel C612 chipset to manage SATA storage – 10x SATA3 (6Gbps)
- Up to 512GB ECC DDR4 2133MHz LRDIMM – 8x DIMM slots (512 Gb RAM with 8x 64Gb mem sticks)
- 1 PCI-E 3.0 x4 (in x8), 1 PCI-E 3.0 x8 (in x16), 2 PCI-E 3.0 x8, 1 PCI-E 2.0 x2 (in x4), 1 PCI-E 2.0 x4 (in x8)
- Intel i350-AM4 Quad port GbE LAN
- 8x SAS3 (12Gbps) via LSI 3008 SW controller; RAID 0, 1, 10
- 1x VGA, 2x COM, 1x TPM
- 4x USB 3.0 ports, 8x USB 2.0 ports
- 2x SuperDOM with built-in power
- Intelligent Platform Management Interface v.2.0 (IPMI 2.0) with virtual media over LAN and KVM-over-LAN support
The Format and compatibility
There is also X10SRG-F which is listed as compatible with ESXi 5.5U2 on the Supermicro website, has the same set of components, but does uses “only” the C612 chipset for SATA drives. So there is no SAS with an LSI 3008 which provides SAS data transfer rates of 12, 6, and 3Gb/s per lane, and SATA at 6 and 3Gb/s data transfer rates per lane. But the format is a proprietary format (not ATX).
I wanted ATX format so I could get my own choce for a case. In fact I wanted to get Fractal Design Define R5, Black Pearl, ATX Mid Tower, Case with easy access to disks and easy opening, when testing.
The board has a standard 8x 4pin PWM fan connectors on the front and back sides. 1U, 2U, 3U or 4U/Tower chassis configurations are possible. If you want to build the system by using the Supermicro cases you can, but in this case you'll also have to be ready to pick a power supply at the same time. I picked 92+ platinum powersuply (92% efficiency) from FSP Group.
Expansion Slots and I/O of the X10SRH-CLN4F
Very large number of PCI Express lanes giving options for expansion.
- PCI-E lanes – x4 and three x8 v.3.0, x4 and x8 v.2.0
- USB 3.0 ports – 2 rear, 2 internal header.
- SuperDOM support
- SATA Disk-on-Module (DOM) – The ports are yellow color on the pic. above…
- Serial port (for UPS for example)
- IPMI for remote management
- 4 Intel i350-AM4 gigabit ports
- 1 VGA
Make sure that you get a narrow ILM socket compatible cooler as the square Socket 2011 coolers will not fit! Check the list of compatible hardware on the main page – Building Energy Efficient ESXi Home Lab.
Narrow ILM socket compared to square ILM socket – Image courtesy of Noctua.
The Haswell based platform is third generation E5 that has been announced as low power consumption optimized with large number of PCIe lanes and USB 3.0 ports. Coupling that with DDR4 which runs at 1.2V compared to 1.35 V for DDR3 are faster and more power efficient.
Up to 50% increased bandwidth and up to 50 % increased power efficiency.
The Xeon E5-2600 V3 family does support up to 14 cores per socket and 28 threads coupled with fast 35 MB LLC (Last Level Cache) and having both the Intel Hyper Threading and Turbo Boost technologies.
Update: The Low power E5-2630L v3 (which I chosed) and E5-2650L v3 run at 1.8 GHz. They have 8 and 12 cores, and they have 55 Watt and 65 Watt TDP. However the latter one is inaccessible due to the high price.
System build around this architecture will cost you less on a long run when used at home 7/7 as a storage or virtualization server. The platform shall still be relevant not in 1-2 years from now on but perhaps 4-5 years from now on as well.
Update: The built-in LSI Logic Fusion-MPT 12GSAS SAS3008 PCI-Express which is on VMware HCL. The LSI 3008 is 12 GbpS SAS where the H310 I used previously is 6Gbps only… You can check yours too – how to check queue depth of your storage adapter
Update: New esx cli commands shows also which controllers are used by vSAN cluster and what's the queue depth. It has been added in vSAN 6.6 (ESXi 6.5).
Just connect to any ESXi part of the vSAN cluster, via putty, and enter:
esxcli vsan debug controller list
You'll get this view…
Energy Efficient ESXi Home Lab:
- Efficient Home Server – Start with an Efficient Power Supply
- ESXi Home lab – Get a quality and comfortable case
- Supermicro Single CPU Board for ESXi Homelab – X10SRH-CLN4F – this post
- Supermicro Single CPU Board for ESXi Home lab – X10SRH-CLN4F – Part 2
- Supermicro Single CPU Board for ESXi Home lab – Upgrading LSI 3008 HBA on the X10SRH-CLN4F
- Building Energy Efficient ESXi Home Lab – parts list [this post]
- Homelab – Airflow Solar System
From a Wife Acceptance Factor (WAF) perspective, what does that sort of build come out as in terms of cost?
It depends on how much RAM you pick (8, 16, 32 … or 64) per stick… You can go quite through the roof if you maxing the RAM. The built is a bit over €2000, but you can go cheaper way with the same mobo or with a mobo without the LSI 3008 and less built-in LAN ports. (ex. the X10SRI-F-O)…
Wonderful website and expertise for an old and I mean old geek (67) like me. I am about to go with the X10SRI-F and am hoping I won’t have problems with ESXi 6.0. I’ll use the E5 lowest wattage chip. Am doing nested Esxi currently on I7-2600K 32GB with autolab.
Also am thinking about the Avoton C2700 supermicro board.
Right now am running 3 hosts under esxi 6 + VCenter and VMturbo and DC on outer lab. Seems sluggish and the keyboard/mouse is difficult as I only have a mac laptop + server.
Best
Don
Hi Don
Im about to get the same board (X10SRi-F). Any issues with ESXI 6? Does it work straight out of the box with drivers etc?
Cheers,
James
Yes. The only think I needed was a driver for my 10GbE Intel NICs, but those are not part of the board itself..
Nice new lab you’re a bout to build up.
Personally I went for a Supermicro X9SRH-7TF with onboard 10GbE and onboard LSI 2308 (QD of 600). The good thing is everything is on VMware HCL!
One question though why you are going for a E5-26xx series v3 (dual socket) and not the cheaper E5-16xx v3 series?
Hi Didier,
Yes I know that board -:). @ErikBussink uses it in his lab too. Nice to have everything on the board.
I wanted to go with the new platform to be able to build more energy efficient system. And that’s why also I went for the E5-2630L v3 CPU which has only 55W TDP compared to E5-16xx which has 140W TDP. As you say the E5-16xx are cheaper, but the power consumption is higher.
You have a very good point there about processor power consumption!
Note that about IPMI, you’ll need a licence to activate the remote console/virtual media 🙁
Didier,
Just to let you know.
Mounting ISO, remote console/virtual media works OOB. Without special license…
Best,
Vladan
Hi, I’m actually thinking of building a system around the same board for a vmware server. Is everything working well in vmware for it? Also, I know that usually sas controller are backward compatible with sata. Is that the case with the LSI controller on this board, I really don’t want to use the raid on the c612 chipset.
All the necessary parts haven’t showed up yet. I’m waiting for the delivery. But yes SAS is backwards compatible with SATA so there should not be a problem to attach SATA drives to the SAS/SATA LSI 3008.
What is the expected power usage of the total system ?
Currently my setup draws about 150W per server (2 servers) resulting in a total cost of 1,5 euro per day. Over a period of 2 year, that means 1100 euro of electricity. If I could reduce this to less then 100W and everything on a single server, that could save me at least 700 euro
It really depends of the workloads, and also how many periphericals are connected to the MB. (Ex. 1 spinning disk takes something between 7-9W)….
let me rephrase 🙂
how much is your current setup using on a daily basis and perhaps also interesting how many drives are in your system.
I have an X9SRA setup which uses around 78 watts idle with 13 VM running and using 45 GB / ram, disks sleeping. The spec is:
1x E5-1620 V2
8x 16GB Quad rank DDR3 ECC (128GB)
9x 6TB WD red (sleepmode)
4x Samsung SSD 850 Pro 500GB
1x LSI 9211-8i (M1015 reflashed to IT-mod) passthrough to fileserver and connected to 8x6TB WD Red.
If I remove the LSI card and shut down all VMs it Idles around 52 watts with ESXI 5.5 idling.
The PSU is Supermicro PWS-441P-1H 440W/480W … Platinum. I do wonder why its speced 440/480. Most PSU have one spec.
ESXI 5.5 is set to low power mode. The thing I need to change is the switch. Cisco SG-300-28P, which uses around 26 watts. There are switches out there that uses 6-7 watt without POE which I am not using at the moment. So with this switch it uses slightly over 100 watt with everything running.
Not bad. Yes, the switch can get “hungry” on power as well. And sometimes it’s difficult to chase single box and blame it for power consumption as it turns out that 3-4 boxes added together DO consume some huge power at the end of the month (year).. -:)
So it’s really difficult to build a lab which IS energy efficient AND at the same time it’s physical lab with 3-4 boxes + switch(es)…
A choice must be made before or a compromise of both is the way to go. One must say, ok, I have a lab which consume on a long run, but at the same time I got the work done! In my situation when all the hyper-converged solutions are emerging on the market, to be able to test them, I clearly can’t go with nested lab. But at the same time I try to invest into something that consume twice (or 3 times) less than current v2 systems….
Do you have the full details on your complete setup? I was using this as a guide, and I’m ready to pull the trigger…but the memory options can be dizzying; I haven’t built a server for a vmware environment in a long time (have mostly used work lab equipment). I’ve been using vagrant a lot more for vm development, and thought it would be good to have a home server to do development on. Thanks for your tips; great research! I’m mostly hung up on RAM (64GB or higher) and about as small of a footprint as possible (case-wise) for an ATX MB.
Hi Vladan,
I’m thinking of investing in the following lab as it will be sponsored by my company.
Before i go for the nested lab, what are the pros and cons of the physical lab.
my nested lab specs are as follows, shown below. I also have the same CISCO switch that you are using. I will mostly be doing NSX stuff.
Would appreciate if you could share your wisdom.
Cheers!
Mo’
Processor
2 x Quad-Core Intel® Xeon® Processor E5-2603 v2 1.80GHz 10MB Cache (80W) Processor Heatsink
2 x Thinkmate Quiet Cooling Performance Heatsink for Intel LGA2011
Motherboard
Intel® C602 Chipset – Dual Intel® Gigabit Ethernet – 8x SATA
Memory
16 x 16GB PC3-14900 1866MHz DDR3 ECC Registered DIMM
Chassis
Thinkmate® VSX-R5 Series Full Tower Chassis – Extended ATX – Titanium Grey
Power Supply
Corsair Professional Series AX760 – Platinum Certified – 760W Modular Power Supply
Hard Drive
4 x 128GB Micron M600 2.5″ SATA 6.0Gb/s Solid State Drive 5.25″ Bay
Samsung 24x DVD+/-RW Dual Layer (SATA)
Video Card
PNY NVIDIA® GeForce® GTX 970 4GB GDDR5 (1xDVI, 1xMini-HDMI, 3xMini-DP)
Network Card
Intel® Ethernet Server Adapter I350-T4V2 Quad Port (4x RJ-45)
Intel® 10-Gigabit Ethernet Converged Network Adapter X540-T2 (2x RJ-45)
Operating System
Microsoft Windows 8.1 Professional (64-bit) with Recovery Partition (NON-OA)
Wow, a monster lab. But why don’t you go with latest Xeon E5-26xx v3 and DDR4. (If you can from the budget perspective). It runs cooler with more perf.
The specs are for nested-lab? If yes the 10Gig NIC isn’t really needed IMHO. The SSD, I think it’s better to go for at least 256Gb capacity flash drives as they often has better specs over the 128Gb ones.
Thanks. the lab is sponsored by my company 🙂
so having 10 gig will not make much of a difference as i can certainly add it later on. for the NAS, i already have synology ds 415+ with ssd caching.
what kind of firewall do you recommend for home lab ?
Hi Vladan,
On the basis of your recommendation, i started the process of developing my own rig for the nested lab for both NSX and DC tracks.
Since I already had the budget pre-approved, now its time to get my hands dirty and develop a host based on Xeon E5-26xx v3 and DDR4.
I will keep you posted on how it turns out.
One question I have is the will I be able to run all the scenarios for DC and NSX track while running a nexted monster host or should I look into developing a 4 host lab.
Looking forward for your speedy response.
Cheers!
Mo’
Nested lab is way cheaper, but I don’t think you can use it as a test bed for anything. It’s good enough for learning… You get the point.
Hi Vladan, I plan to build new server for vSphere with Supermicro motherboard. I don’t know if VMWARE can work with the LSI 3008 SW controller. Is there really only software RAID LSI 3008 ? It means that the RAID runs after install some OS ? Runs vSphere 5.5 or 6.0 with this software RAID ? And what about performance ? Please give me advice. Thanks Pavel
Pavel, I shall have those boards to arrive weeks ago…. But due to some (stupid) regulations on Lithium batteries my package returned back to depot twice in the US… It’s a pain ….. However I shall be having one board showing in couple of days….
Concerning the LSI 3008 I’ll not be using it for RAID but pass-through (VSAN) or RAID0 (depends how it goes..). It’s 3rd gen 12Gb/s SAS controller. I don’t think that performance shall be deceiving… The limit will be on the SATA interface (6Gb/s) I’ll be attaching to it anyway…
Hi Vlad,
So in terms of using nested lab by running on a monster host with 256 gig ram and 5tb of ssd storage. just for the preparation of vcix dc, will I be able to test all the blueprint stuff, like cover around 90%. I would really appreciate if you could provide me a quick answer on that as I’m about to pull the trigger on my end. I really don’t want to use it as a test bed. It just going to be used for exam preparation for vcix dc and nv
At this stage I can only get approval for one host from my work place. May be next year I can add another host and a 10 gig switch.
Cheers!
Colin
Wow, 256 gigs of RAM… You can do lot’s of stuff with that. I would say you’re definitely fine for your exam.
Thanks mate!
I will keep you posted about my final build.
Cheers!
Colin
Hi Vlad,
I was wondering if you could provide your build break out for the lab you are currently building from supermicro MOB. I really would like to compare and see the cost.
Cheers!
Colin
Bonjour or Hi , dont know if you want us writing in english :p
One question : where did you get the parts? I’m quite interested in building a similar machine (or two..) 🙂
thanks a lot !
Long story short. In US. But I would NOT recommend that anyone in Europe as the shipping outside to US is a hell. The problem is with new administrative directions (2012 I believe) prohibiting the battery holding the BIOS data, which sits inside of the Mobo, to be shipped outside of US (DHL, FedEx, USpostal…). I believe it’s because the aviation anti-terrorists …
The directives are difficult to understand, so what I did is to I let shipito (i use shipito.com for saving on re-packaging) to take the batteries out of the mobo, and I’ll buy new battery when the mobo will show up. (by the end of next week?)….
There is more in the pipes… -:)
The only reason not to speak french in the comments is simple – only the french speaking understand..-:)
Le seul des raison de ne pas parler francais sur le comments est tres simple – seulement les francophones comprend. -:)
ahah okay thanks! interesting approach :p
Hi Vladan, really nice post and a great build. I was starting to make plans for my own build on Supermicro and after some long time I came to the conclusion that the mobo you have used is really perfect for this! 😀
I wonder if you have received all the parts and if you are already running this build? How does those two controllers work with ESXi and what about everything else? Any problems?
Supermicro website says that this mobo is compatible just with 5.5U1 and not with U2. I did not find it on VMWare ICL. Which ESXi did you use? Perhaps tried also already new version of ESXi 6?
I would like to go with 4x 500gb SSDs in RAID 10 as my main VM storage and with 2 HDD of 4TB spin drives in RAID1 for storing snapshots, backups, ISOs, VM copies and other “dumpings”. It will be only possible to make RAID 10 and RAID 1 on SAS LSI 3008 controller. What do you think?
If required I will get an additional RAID controler.
I will go with INTEL Xeon E5-1650 v3 3.50GHz 15MB 6C/12T CPU and 32 GB of RAM for start.
I was considering to pass through C612 “fakeraid” mobo controller to one of VM and play around with it for FreeNAS, RAIDz, etc. but I believe that this is not possible to do…
The MB runs with latest ESXi 6.0, and VMware VSAN All-Flash. Added a low consumption Intel Ethernet Controller X710-da2 with SFP+ connectors for 10GbE. Check this post where I’m testing the VSAN health check plugin… http://www.vladan.fr/vmware-vsan-health-check-plugin/ – just a slight driver difference making one component “yello” instead of green. If not, it’s fully HCL …
This is a bit of an old post, but I thought anyone reading this might like to know there are issues running the Supermicro X10SRH-CF with ESXi 6.7; when using a hard drive via ESXi’s vmdk, the drive’s data will appear to be corrupt on Windows systems.
“when using a hard drive via ESXi’s vmdk” I don’t think this is something to do with the Motherboard. Unless you do some crazy passthrough?
To briefly recap: when I run Win 2012 R2 on bare metal, I don’t have any problems with large hard drives. And when I run it on ESXi 6.7, the drives are fine when they’re accessed directly by the OS through a different controller (like a 9200-8e with splitter cable) via PCI passthrough, but keep corrupting data when attached to the onboard SATA connectors and accessed through ESXi’s vmdk. I’ve tested this thoroughly and the problem is repeatable.
When I run Win 2012 R2 on ESXi 5.5u1 (which is the latest version of ESXi that SuperMicro says is compatible with this board) I don’t have the same problem. But I do find Windows hangs if I enable pass through to the onboard SAS 3008 :-p
Personally I would not use passthrough. The difference in performance is minimal, and, if used with Flash storage, I don’t really see a reason for passthrough. I remember I was running vSphere 6.7 (even if the board says it’s unsupported), with 3 host configuration and VMware vSAN, just fine. But no passthrough. The on-board SAS 3008 was flashed with IT mode firmware (which is not for RAID, but for vSAN, ZFS etc…) and everything worked including vSAN did not complain about the controller as the controller was on VMware HCL at that time -:)…
Hi Vladan,
I have a question about the memory which you are using in your X10SRH-CLN4F lab. The majority of the CPUs that you listed officially only support DDR4 1600 or 1866 RAM (http://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/xeon-e5-brief.pdf) , however the only certified supermiro memory sets which are validated are 2133 chips.
I assume that you used 2133s in your installation, just wanted to know if you had any issues with compatibility?
Best Regards,
John
It’s the 32Gb model – Supermicro Certified MEM-DR432L-SL01-LR21 Samsung 32GB DDR4-2133 4Rx4 LP ECC LRDIMM Memory
http://amzn.to/1IVP9uW
No problems as It was recommended memory by Supermicro for my mobo.
Best luck with your lab.
Hello, Vladan! Can you advise, please, some Supermicro platform for ESXi 5.5u2 and VMware Horizon View infrastructure? Easy expandable (may be some FatTwin?), with support Samsung SSD’s, Raid 50 etc., 2600v3. I understand it noob question, but really confused in Supermicro. Thanks!
I’d advice to check compatibility. VMware HCL and (or) Supermicro has the compatibility listed also in their detailed specs. Especially if you’re planning to uses it in production environment.
Best luck
Very Interesting article, will definitely influence my choice for a ESXi homelab.
But, depending on your screen size, i am using a 12 inch laptop at the moment, the social media icons which flow up and down on the left of the screen are blocking text which makes un-engaging and it difficult to read.
What resolution are you on OMG. I’m on 1366×768 and feels just ok. You must have then even lower resolution… That’s bad. I’m going to change my laptop because it’s does not cut it anymore…
Hi, vladan need an expert advice
i’m planning to buy Supermicro SuperServer 5028A-TN4 w/ 32gb ram and run esxi 6.0 on it
I’m planning to install maybe 4-8 guest OS on it.
Is Intel® Atom processor C2758F enough?
TIA
Don’t know this CPU. But I doubt it… http://www.cpubenchmark.net/cpu.php?cpu=Intel+Atom+C2758+%40+2.40GHz
I am planning a very similar home build. I am wondering at what point it makes sense to go with a dual processor system? I’m just getting started and I want to do a lot with nested, but I’m not sure where it will all take me. I’m going to be in a position for a one time monster build that I’m trying to future proof for whatever I decide to get into. At the same time, I do need to keep costs low including ongoing power. I’m not sure what to look at to evaluate one or two processors?
Depends on what you’ll be using the system for, consider get storage as fast as possible ( NVMe SSD ?) and lots of RAM…. With all nested you’ll be losing perhaps 40% of CPU power on nested workflows. I have 3-4 boxes running all the time so being on low-power CPUs is a must… In addition, consider heat. The systems I built do not generate almost any (except the 10GbE switch), in case you’re planning to have them in the same room you’ll be working.
Good evening all,
My name is Tony and i a recommendation . I am looking to build a ESXI 6 home lab my budget are limited to about 600.00 US dollars. I understand i will not get a monster box with this amount of money. This is is what i have so far. G.SKILL Ripjaws V Series 32GB (4 x 8GB) 288-Pin DDR4 SDRAM DDR4 2400 (PC4 19200) Intel Z170 Platform / Intel X99 Platform Desktop Memory Model F4-2400C15Q-32GVR. Basically looking for a good Motherboard CPU combo . Trying to stay efficient on the wattage . What CPU would you recommend also a motherboard. I do have plenty of ssd drives Evo 850 Samsung x4 250 gb plus a couple 1 TB storage drives. As far as case and PSU i have SeaSonic X 850 Watts. Thank You very much
Any advice and recommendations are welcome. Sorry my English is so so
Great discussion. I’m very curious as to your thoughts on the Supermicro Xeon-D motherboards and the X10SDV-8C-TLN4F in particular. 8 cores, 16 threads, 45W, 128gb memory max with 2x 10GbE and 2x 1GbE with 12 and 16 core variants on the way. The 5028D-TN4T system (using that same motherboard) looks tailor made for home labs.
Hi Vladan, I’m looking to build a similar esxi system based on what you have here. I am just wondering if you know of a newer supermicro board other than the X10SRH-CLN4F that you used in your system as this is now 18months old? The Supermicro boards available in Australia is limited, for this reason would love to be living in the US.
Well, not really as my system works good so far and I’m not planning to add an additional host. Still have plenty of space for upgrades of RAM (only one host has 128Gb). Can grow up to 256 (see 512Gb). I haven’t checked back for newer version and the compatibility with latest ESXi. Had to pick custom VIBs for my 10GbE Intel NICs and for the built-in storage driver (VSAN requirement). Now I have all green checks ok.
Well, I had to buy mine in US and have shipped it with a mailing service called Shipito. Here at Reunion there is no local representative of Supermicro. Other guys such as Fujitsu, Dell, IBM are… but I wanted a hosts wit iPMI
Best luck and if you succeed, please come back to share.
Hi Vladan,
I ended up buying the Supermicro X10SRH-CLN4F and the Define R5. My first impression of the case is good. However I’ve come to install the motherboard and when I trace out the mounting holes, there is no corresponding hole in the R5 case to install the standoff for the mounting hole that is at the edge of the motherboard to the left of the second 4 memory slots (next to JPI2C1 – Power SMB). It looks like the hole in the R5 case would be under the memory slot. Did you have this issue as well and just not install a standoff for that mounting hole?
I also bought the same cooler as well. Did you need to change the mounting on the back of the motherboard? It looks as though I can just mount it directly into the motherboard as is. Lastly, did you need to change any of the jumpers on the board or was everything fine as it came from the manufacturer?
Thanks,
Thorsten
Thorsten,
I don’t remember for the hole. Frankly, it is possible.
The cooler’s only problem I had was the fact that I had to move slightly (1cm about) the fan higher, to allow the memory slot to be populated. The memory slot which is close to the CPU. But then I just pushed down on the fan back down, to touch the memory, now in place.
No jumpers change, but make sure you read the latest manual corresponding to your motherboard build. Things changes over time.
What I like with this board that there is large capacity for future upgrades. I’m currently running 128Gb (with 4x 32Gb slots filled).
I plan to install esxi ,will the vga have output if I don’t use a cpu without any video card ?
If you don’t use CPU? Explain.
Hi vldan
Great article and reference point even after all these years absolutely awesome.
Just had one question at this point while start this journey.
I can’t seem to find a decent website in Europe UK that would deliver a super micro board , Google rates senetic.co.uk with high ranking but they are turning out to be absolutely useless as I can’t even get a response to buy 3 * X10SRM-TF or X10SRM-F micro ATx boards currently listed @ between circa £280 -£380 price point depending on whether I include 10gbe
The SoC Xeon D15xx ITX SFF boards are working out too much for me sadly and any X99! consumer boards I think would not support ECC type memory that I want..
It’s the one time I wished I lived in the US, as i really want start this lab, hoping either you or someone on here can point me to some trustworthy sites here in UK or Europe with quick delivery
Thanks in advance J
Regards
Thanks. You can use shipito.com or similar service to deliver to the UK. That’s what I’ve done to deliver to Reunion Island…. You basically let them ship to their depo. Then they repackage (group) and send it to your place.
Sorry I haven’t done any research on hardware lately so I won’t be any help here.
Hey Vladan,
Thanks a bundle for your quick reply, appreciated as its an old article but still a great boon for many I’m sure..
One last question I take it you sourced the SuperMicro MOBO directly from Supermicro website or reseller in USA ? and then used shipito, I was considering MyUSA, but will check out shipito as sell if I can a board first :-)..
I was hoping to know what site to use for the MOBO so that I may focus my efforts there..
Regards
If I remember right, the mobo i found at that time, was found at ebay and SuperBiiz.com. But that was long time ago… Wish you good luck with your project. Supermicro is a great choice, especially with the iPMI integration, on their boards.
Also how is your lab after all these years still future proof then? Worth the journey?
Thanks
Actually, I sold 3 out of 4 hosts because I moved to a small apartment (keeping the 4th for W10 with VMware Workstation, 128Gb of RAM and nested labs). Those 3 hosts are doing fine. One is running ESXi 6.7 and the two others hosting replicas at a secondary site.