Yesterday I installed 5 new ESXi3.5 host at a client site, installed VirtualCenter and VMware Update Manager. I configured all hosts with storage, networking, licensing and ntp settings. All looked fine and ready to rock. One last action before starting with our test plan was to update all hosts to the latest patch level and that’s where the ‘fun‘ started.

I ran the task to download the latest patches and scanned the datacenter for updates. Result 5 out of 5 hosts were not compliant to the default baselines. So I initiated a remediate action to update all hosts. All went fine and the first two host updated without a glitch then the third host was skipped and VUM started with host four and also skipped the fifth host. So the first remediate resulted in 3 updated hosts and 2 failed hosts. After several attempts to retry the remediate action without success we decided to flash the USB with the HP image from the VMware website.

After the USB flash we configured the hosts again and initiated the update sequence again and all seemed to work fine. The third host updated without problems and the fifth started fine so we went home hoping to see five up-to-date servers this morning.

WRONG!

The fifth host had started updating but stopped half way through the process with a ‘fault.PatchIntegrityError.summary‘-error. The first things we tried were a restart of the management agents which did not solve the problem and a reboot which did not solve it either. A look at the console almost gave us a hart attack. The console was full of warnings mentioning cpu’s, linuxmem addresses and UNIMPLEMENTED warnings.
(cpu#:3012)WARNING:LinuxMem:201:UNIMPLEMENTED! (addr=0x#######)
We suspected a hardware issue but on the third host which had the same issues we saw that these events dissapeared after the update ran successfully. So we tried to solve and with success.

We tried several things. The first ‘Google’s‘ suggested a time issue. We checked that but unfortunately this was not the case. Just to be certain we removed the ntp settings and re-applied them just to be sure with no result. Then we restarted VirtualCenter and Update Manager also with no result. We were at the point of flashing the USB with the HP image again but that would be too easy and we hadn’t solved the problem, we worked around it and there would be no satisfaction in that.

Then we had 2 options left. Do a RemoteInstall using these instructions or remove the host from VirtualCenter and adding it again. Luckily we tried the second option first so I disconnected and then removed the host from VirtualCenter and after that was completed I added the host again.

Now the big moment I was waiting for. I started the update process again and WOW the update started and finished SUCCESSFULLY!

SATISFACTION AT LAST!