How to: build an ESXi whitebox

Last week I decided to buy a new lab server and I doubted between a HP or Dell mini server or an ESXi whitebox. Because most mini servers only have 8GB memory, I decided to collect specific parts to build my own VMware ESXi whitebox.

To find parts which are compatible with VMware ESXi 4.1, I used the following resources:

• Ultimate VMware ESX Whitebox;
• VM-help.com.

I chose a AMD Phenom II X6 processor, socket AM3 six core processor because it’s a lot cheaper than the Intel Core i-processors. As the basis I needed a AM3 socket motherboard and my selection criteria where simple, 16GB memory and onboard video.

As an ASUS fan I had to choose between the ASUS M4A88T and M4A88TD. Both can house 16GB of memory and have onboard video but the TD version has SATA 6Gbps. Because storage will most likely be the bottleneck, I decided to go for the M4A88TD-M but on the above sites there was no entry for this motherboard.

But Google is my friend so I searched for ‘M4A88TD’ in combination with ‘ESXi’. I found a few sites which mention an almost identical combination of motherboard, processor and ESXi 4.1. Eventually I took a gamble and ordered the ASUS M4A88TD-M/USB3.

The last two items where the simple ones, two sets of 8GB dual channel memory and a 6Gbps SATA disk.

The total kit list is as follows:

• AMD Phenom II X6 Black Edition 1090T processor (6 x 3,2GHz);
• ASUS M4A88TD-M/USB3 motherboard;
• Corsair 8GB DDR3-1333 XMS3 Dual Channel memory;
• Western Digital Caviar Black WD1002FAEX 1TB, SATA-600 hard disk;
• 1 x Intel & 1 x Broadcom Gigabit network adapter;
• HP midi tower with 450W power supply;
• DVD player.

I had an old solid HP/Compaq tower with an old Pentium 4 in it. So I removed the P4, motherboard, IDE disks and power supply and replaced it with my whitebox hardware.

After bolting, screwing and pluging everything together, it was time to install ESXi, this is where the problems started. I wanted to use 100% of the 1TB Western Digital disk for virtual machines so I installed VMware ESXi on a 2GB USB key. (I will discuss the troubles with the installation in a future article.)

When I finally succeeded in succesfully installing ESXi, booting this was a problem. I experienced several issues from hanging boot sequence on USB controllers and network interfaces. To boot ESXi succesfully I had to disable the onboard network interface, the USB3 controller and C1E.

Finally, SUCCES! My VMware ESXi whitebox was running and I could start installing a new lab infrastructure.

But the most important of all, is it any good? It’s great to build an ESXi whitebox but when the performance of all those ‘desktop components’ suck, it’s maybe better to spend a bit more $$. In short, it’s great, performance is comparable to that of enterprise servers with the exeption of disk related tasks. The disk performance is good but it’s not great. I think I’m just spoiled by all those nice enterpise disk solutions. You just cant compare disk I/O of simple desktop despite the fact that I bought a new, fast, 6Gbps SATA disk. The magic word here is spindles, so I’m planning to add another 1TB disk and create a RAID 0-stripe set. Maybe that will speed it up a bit.

At the moment I’m running VMware ESXi 4.1 with:

• vCenter server;
• Domain Controller;
• vCenter Mobile Access appliance;
• View 4.6 connection server with:
  • Automated floating desktop pool of 2 Windows XP desktops with View Composer;
  • Automated floating desktop pool of 2 Windows 7 desktops with View Composer;
  • Manual floating pool of 1 Windows 7 desktop for my downloads.

CPU load is as expected very low, 4723MHz on average. The total memory load when running all those 9 virtual machine is 8,5 – 9GB! A servers (3) run on 2GB memory, the vCMA has 512MB, Windows XP desktops run on 1GB and Windows 7 on 1,5GB, so hooray for Transparent Page Sharing (TPS)!

All things considered I’m very pleased with my ESXi whitebox, performance is good, 16GB of memory gives me enough space to deploy lab VM’s and the money I spend on it is well within my budget (€450,-).

Hint and tips for those of you who want to build their own ESXi whitebox:

• Research, research, research.
  I still hear people buy incompatible hardware despite the available online resources. I mentioned two of them above, check if your desired configuration has already been build. If not Google is your friend;
• Do not save on your harddisk.
  If you save on your harddisk you will be sorry very soon so find a fast disk or even add a SSD if your budget allows it.
  If your budget is a problem, save on the processor. As you can see, the load on my processor for instance is very low. Buy a quad core instead of a six core processor and spend that on a good harddisk.
• Go for a motherboard which can hold 16GB of memory.
  Even if you do not need 16GB right now, shortage of memory is the second bottleneck you will encounter. Try upgrading to 16GB when your motherboard will only hold 8GB max.