VMGuru

I am going to setup a mobile lab on my MacBook Pro with the help of VMware Fusion 12 so I can run some ESXi servers with nested virtualization. This way I will have a lab I can carry around with me on my travels around the world.

I hear you thinking why do you not deploy the lab in a cloud? I have three reasons for that:

Security - I am mostly testing some fast things with alpha or beta software that also is tied to cyber security in some way and some of those gems you don’t want to let loose in a shared lab environment or a cloud which scans the software running…
Presenting – If I can get it working smoothly, I can start using it on main stage so not relying on that event WIFI together with everyone else.
Costs – I irregular test things, so running a lab in the cloud will cost more money, even if you shut down everything after being done you still have the storage cost ticking. Also, my tension arc is that of a goldfish so I might go down one rabbit hole exploring and find 10.000 other interesting things not necessarily tied to my main objective while keeping the lab running and accumulating costs.

Planning of the mobile lab

For the creation of my mobile lab I will be using several software packages from Apple, VMware, Microsoft and Veeam but the hardware base will be a MacBook Pro 16” (2019).

Software used:

macOS Monterey - version 12.4
VMware Fusion – version 12.2.3
VMware ESXi – version 7.0.3
VMware VCenter Server Appliance (VCSA) – version 7.0.3
Microsoft Windows Server 2022 - June update
Veeam Backup & Replication v12 Beta 2

Hardware used:

MacBook Pro (16-inch, 2019)
CPU Intel Core i9, 2.3 GHz, 8-core
64 GB 2667 MHz DDR4
2 TB SSD

Getting started

I had been running with this idea of a mobile lab for some time now, should be easy right? Let me share my experience with setting it all up and the tips and tricks I discovered along the way.

Step 1 – Installation media

For the VMware software packages, Fusion and vSphere, you can go to VMware Customer Connect Download Page or as I did go to the vExpert portal. You must be a vExpert though, but fortunately I am a veteran vExpert I have always loved these perks of being a vExpert and that helped push my career faster forward. Installation of VMware Fusion Pro is straightforward on the MacBook just follow the wizard and it will be done before you can blink your eyes.

VMware Fusion version 12.2.3 as a base

The first challenge I ran into was that I downloaded an installation iso for VMware ESXi in the form of a gzip file (.gz) but that file cannot be used natively as a source within VMware Fusion to install an ESXi server. It first needs some converting.

VMware ESXi ISO in a gzip file

How to extract or convert a gzip (.gz) file that contains an .iso to a working iso file for installations? The easiest way I found, that works, is by decompressing the .gz file through:

> gzip -d <FILENAME>

decompresses the file and you will get the compressed iso file hiding within.

Step 2 – Installing VMware ESXi in VMware Fusion Pro

Now we have the required ISO for ESXi we can start the installation of two ESXi servers in VMware Fusion Pro. Go to File and select New from the menu. Then drag the newly obtained ISO in the window and start the installation. I will be installing two ESXi 7.x servers with 4 cores and 16GB each.

VMware Fusion Pro 12 - Select Installation Method

TIP: When your mouse gets ‘stucked’ in Fusion use the <^ Control> + <⌘ Command> key to release it from the active installation window.

On the next steps select the operating system, in my case VMware ESXi 7 and later, firmware UEFI (maybe go for the UEFI Secure Boot option) and finish the installation.

The installation of VMware ESXi in Fusion as a virtual machine (Nested virtualization) is quite straightforward these days. Another challenge I had that I needed to press some function keys for controlling the installation and configuration. Just go to the VM window and select in the Fusion menu bar Virtual Machine > Send Key > and the function key you require. In my case it was F11 which is tied to other MacBook actions.

Step 3 – Deploying a vCenter Server Appliance (VCSA) on top of the ESXi

Now the base of the lab has been setup, lets deploy a VCSA on top of an ESXi host. Just mount the downloaded VCSA iso and go to the folder [vcsa-ui-installer > mac > installer] and press the Deploy vCenter Server button on the screen.

Deploying the vCenter Server Appliance (VCSA)

I ran into a next challenge here that VMware is seen as not verified and an error is thrown installer cannot be opened because the developer cannot be verified. This behavior is due to a security change in MacOS Catalina. You can try to use <^CONTROL) click instead and open from the menu it pops-up but you will get several errors in a chain. There is a better and faster way. We will have to modify the security settings on the Mac until the VCSA deployment finishes.

Open a terminal window and use the following commands:

> sudo spctl --master-disable

> put in your Password to activate the command

> xattr -r -d com.apple.quarantine <FILENAME>

> sudo spctl --master-enable

> put in your Password again to activate protection again

Modifying MacOS Catalina security settings for spctl

spctl manages the security assessment policy subsystem. This subsystem maintains and evaluates rules that determine whether the system allows the installation, execution, and other operations on files on the system.

NOTE: For the deployment of the VCSA for a tiny environment, like a lab, you will need at least 589 GB disk space for version 7 (this was 300 GB with version 6.7)

Step 4 – Configuring vCenter Server Appliance (VCSA)

The raw foundation of the lab is almost complete now, but when I restarted the VCSA after the installation finished, I ran into yet another challenge. The combination with Google Chrome, MacOS Catalina and the security settings on default doesn’t trust the self-signed certificate of the VCSA. You will get the message Your connection is not private and as error NET::ERR_CERT_INVALID. An easy way around this is by typing the text:

thisisunsafe

when you are on the error page, and it will redirect you to the vCenter UI page. The bypass adds an exception for that particular domain to chrome’s internal memory. You can remove the exception by clicking on the padlock icon and re-enable Warnings link. Sometimes after an chrome upgrade or, reboot of the Mac you might need to retype the thisisunsafe phrase to add the exception again.

Chrome - Your connection is not private - NET::ERR_CERT_INVALID message

You can also fix this by replacing the default vCenter certificate with a free Let’s Encrypt SSL certificate. How to do that you can read in this easy-to-read blog of Virtuallywired.

Step 5 – Post-Configuration changes

While the installation went on, I have dived into the message I got at the beginning when installing the ESXi servers on a VM within VMware Fusion Pro. From security perspective it is better to leave this enabled, but this is a lab and will never run production workloads so let’s switch it off and gain a few percentages extra performance.

To disable side channel mitigations, make sure the installed ESXi server VMs are not running and open the virtual machine settings by selecting the VM in Fusion and in the Menu Bar > Virtual Machine > Settings or Right Click with your mouse on the VM and choose Settings from the dropdown menu. For more information, please see this VMware knowledge-base article KB79832

Conclusion

The mobile lab is running smooth now on my MacBook Pro with better than expected performance. I am now deploying Microsoft Windows 2022 Servers and running security tests with Veeam Backup & Replication v12 Beta 2 in combination with gMSA, Kerberos, NTLM, Hardening and Active Directory management domains.

Overview of VMware Fusion with nested ESXi 7 Servers and a VCSA on top