I have had several customers asking me for advice on what to do with new ESX hosts who should be joined into the VMware cluster, but after adding to the cluster problems with vMotion arose. It just didn’t work anymore because of some minor CPU differences.

One of the customers had bought the exact same HP DL380 model with same product number and revision with the same type of Intel CPUs inside. But unfortunately the stepping on the old CPUs is 6 and on the new ones its 10. The HP machines contain Xeon CPUs from Intel type 5450 3 Ghz stepping 6 (existing) and stepping 10 (new).

I asked them the following questions so I could give them advice on what to do next:

Question: How many new ESX hosts are you adding to the cluster?
Impact: If you are just adding 1 or 2 ESX hosts as extra capacity it is good to look into reforming the VMware cluster to an EVC cluster., because the more ESX hosts (up to 32) in a DRS cluster the better DRS can do its job. If you have more than 2 hosts to add to the cluster it can be a solution to build a dedicated cluster with the new ESX hosts.

Question: Do you need the new features added to the CPUs or do you just need more power in the VMware cluster?
Impact: If you look at the latest range of CPUs it can make a total difference of up to 25% speed, because of new added features.

Question: What are the plans for the future with the clusters and do you suspect significant growth?
Impact: If you suspect significant growth it can be useful to build up a new ESX cluster with new functionality but always weigh carefully the pros and the cons.

Question: Are the used servers and CPU’s capable of switching on the VT or AMD-V option and can the XD or NX bit be enabled in the BIOS? (Intel markets the feature as the XD bit, for eXecute Disable. AMD uses the name Enhanced Virus Protection.)
Impact: If the machine and the CPUs are capable, you can start using an VMware EVC cluster.

After answering the above questions I recommended some clients to build up an EVC cluster in vCenter Server, most answers I get after suggesting such a move are:

“uhhh build an EVC cluster, what’s that?”

VMware Enhanced VMotion Compatibility (EVC)—available in VMware Infrastructure 3 beginning with version 3.5 Update 2—facilitates VMotion between different CPU generations, taking advantage of Intel Flex Migration and AMD-V Extended Migration technologies.

When enabled for a cluster, EVC ensures that all CPUs within the cluster are vMotion compatible. CPUs starting with Intel 45nm Core 2 (Penryn) and AMD Second Generation Opteron (revision E or F) incorporate FlexMigration and Extended Migration technologies, respectively.

The EVC feature allows the virtualization layer to mask or hide certain features by modifying the semantics of the CPUID instruction and hides certain CPUID feature bits, even from nonprivileged code. For example, with support from hardware, the virtualization layer modifies the semantics of the CPUID instruction to mask or hide the SSE4.1 feature from any code (privileged or nonprivileged) to make CPUs differing in this feature compatible for vMotion. Specifics on CPU compatibility with the Enhanced vMotion Compatibility feature

It’s recommended to use CPU masking on cluster level instead of virtual machine level whereby CPU masking can be done on cluster level. You will have to build an EVC cluster though, possibly limiting the total power of a VMware cluster.

Boundary conditions for Enhanced VMware Cluster (EVC) are:

  • CPUs from a single vendor, either AMD or Intel
  • Running ESX Server 3.5 Update 2 or newer
  • Connected to vCenter Server
  • Hardware support for AMD-V or Intel VT and enabled in the bios.
  • AMD no execute (NX) or Intel execute disable (XD) technology
  • Support hardware live migration AMD-V Extended Migration or Intel FlexMigration.

After explaining the whole EVC cluster principle you can guess the next question in line I suspect. “How do we build up an EVC cluster?”

Steps to take to build an EVC cluster:

Step 1
Build a new VMware cluster with the new host(s) and make sure the NX bit is enabled in the bios. Also switch on VT or AMD-V technology in there. No VMs may be present or switched on, on the ESX host. DRS and HA technology is fully compatible with a VMware EVC cluster.

Step 2
Change the VMware EVC mode in the cluster settings from disabled to enabled.

EVC Setting

Step 3
Bring down the VMs on an ESX host you want to move into the VMware EVC Cluster. Use a cold migrate to move them to the new EVC cluster. Start the VM’s on the EVC cluster. After moving all the VM’s from the old ESX host, restart the host and edit the BIOS settings for it to join the EVC cluster. Add the ‘old’ ESX host to the new EVC cluster.

Step 4
Repeat step 3 for every ESX host you want to add to the EVC cluster. Also look on page 238 in the Basic System Administrators Update 2 and later for ESX 3.5, ESXi 3.5 with Virtual Center 2.5. (vi3_35_25_u2_admin_guide.pdf)

Background info:

For AMD Opteron there are two VMware EVC mode options to choose from at the moment, namely:

AMD Option 1
Applies the baseline feature set of AMD Opteron™ Generation 1/2 (“Rev. E”/”Rev. F”)
processors to all hosts in the cluster.

Hosts with the following processor types will be permitted to enter the cluster:
AMD Opteron™ Generation 1/2 (“Rev. E”/”Rev. F”)
AMD Opteron™ Generation 3 (“Greyhound”)

AMD Option 2
Applies the baseline feature set of AMD Opteron™ Generation 3 (“Greyhound”) processors
to all hosts in the cluster.

Hosts with the following processor types will be permitted to enter the cluster:
AMD Opteron™ Generation 3 (“Greyhound”)

Additional CPU features exposed include SSE4A, MisAlignSSE, POPCOUNT, ABM (LZCNT)
and CMPXCHG16B.

For Intel there are three VMware EVC mode options to choose from at the moment, namely:

Intel Option 1
Applies the baseline feature set of Intel® Xeon® Core™2 (“Merom”) processors to all
hosts in the cluster.

Hosts with the following processor types will be permitted to enter the cluster:
Intel® Xeon® Core™2 (“Merom”)
Intel® Xeon® 45nm Core™2 (“Penryn”)
Intel® Xeon® Core™ i7 (“Nehalem”)

Change to Intel EVC mode

Intel Option 2
Applies the baseline feature set of Intel® Xeon® 45nm Core™2 (“Penryn”) processors to all
hosts in the cluster.

Hosts with the following processor types will be permitted to enter the cluster:
Intel® Xeon® 45nm Core™2 (“Penryn”)
Intel® Xeon® Core™ i7 (“Nehalem”)

Additional CPU features exposed include SSE4.1.

Intel Option 3
Applies the baseline feature set of Intel® Xeon® Core™ i7 (“Nehalem”) processors to all
hosts in the cluster.

Hosts with the following processor types will be permitted to enter the cluster:
Intel® Xeon® Core™ i7 (“Nehalem”)

Additional CPU features exposed include SSE4.2 and POPCOUNT.

Interested in knowing if all your physical ESX servers are the same? VMware CPU Host Info will help you find out. The application gathers the important system information from your hosts and puts this in one single overview. This tool is written by Richard Garsthagen.

References
See the following references for more information on EVC clusters: