VC Upgrade went haywire

Yesterday was a fun day. What looked like a simple, clean upgrade became something more interesting. After making a before snapshot I started the VirtualCenter installation wizard. Then the fun started. Nothing happened.

Let me rephrase that. NOTHING happened. Machine hung. Clicking the reboot button in the VI client caused the machine to reboot. At least, so it seemed.

The machine didn’t get back to life. VirtualCenter thought that it was in limbo, not started and not shutdown. So I restarted the management services on the ESX host, you know, service mgmt-vmware restart and I even restarted the VirtualCenter client by hand like service vpxa restart

It looked like the problem got bigger. VirtualCenter now saw the machine as invalid. Logging on the ESX host with the VI client it looked very very strange.

After trying the usual stuff like

vmware-cmd /vmfs/volumes/Production1/SRV01/SRV01.vmx stop hard

I went for the more rigorous approach. I tried unregistering and registering the VM with the host like:

vmware-cmd -s unregister /vmfs/volumes/Production1/SRV01/SRV01.vmx
vmware-cmd -s register /vmfs/volumes/Production1/SRV01/SRV01.vmx

Even this didn’t work. FUBAR as they used to say in the telco business.
Since I suspected a host issue I tried unregistering the machine from this host and registered it on another ESX host.

Everything looked great again. The machine was visible on the ESX host. I carefully started to breath again.. Too soon..
Dang! “Could not power on VM: No swap file. Failed to power on VM”.

Knowing the swap file was present I suspected the files were still in use by the original VM.
While googling I came across the site from Rene Helweg , with whom I worked in the past for a client in The Netherlands.

ps aux | grep

Strangely enough ps didn’t give anything. No rogue processes. Hmm.. gonna be a long day.

After consulting some pals (TAZZ) I tried the vm-support approach Ifound on the blog from Sanjat Kabi.

the vm-support -x gave a list of all worlds. My hung machine was in the list.
with vm-support -X it does some really nifty things. Not sure what it does exactly but it said: “This might crash your VM’.. well, it is broken, who cares. After that it was ‘This will crash your VM’. Fine :D

It ran for a while and.. whadayaknow.. It worked!! VM crashed so I started it. It was fine!
Still in the mode of updating VirtualCenter I started the installer again… Wheeeeee!!!! Running. Finally the machine was upgraded to Update 3.

Update
Just got the client on AIM. Their exact words “The customer was happy .. yippee :-P”