VMGuru

Power Outage When running a virtual infrastructure based on VMware vSphere, you have multiple techniques to create a high available environment. You can create a cluster, use VMware HA or FT but when the power fails you’re done. To buy us a little bit of time, you can add a UPS with enough capacity to power your servers, switches and storage for a limited period of time. Just enough to start a standby generator or just wait until the power returns.

But what when this takes too long and you must power down your virtual environment including all virtual machines?

When you’re on-site and aware of the power failure you can shut everything down manually of course but what when you’re not there, in the evening, during weekends, etc.? You return in the morning finding that all your virtual machines are down and corrupted?

You will have to automate the shutdown of the virtual machines and ESXi hosts and I found two ways to do this.

Option 1 (APC UPS):

7 out of 10 UPS-es I come across are APC UPS-es and APC has some nice management software to automated server shutdown. To automate the shutdown of your virtual machines and ESXi hosts you will need:

APC PowerChute Network Shutdown (PCNS) 3.0.1;
VMware Management Appliance (vMA) 5.1;
TCP ports 3052, 6547, 80 and UDP port 3052 need to be open on the network.

First, install and configure the VMware Management Appliance on your VMware vSphere infrastructure. This is a basic VMware vSphere task, so I won’t bother you with details or screenshots but it comes down to this:

Connect to your VMware vSphere environment using the vSphere client software or the vSphere web client;
Select the host to which you want to deploy vMA in the inventory pane;
Select Deploy from a file or URL if you have already downloaded and unzipped the vMA virtual appliance package;
Click Browse, select the OVF, and click Next two times and accept the licence agreement;
Specify a name for the virtual machine;
Select an inventory location and a resource pool where the virtual machine will be placed;
select the datastore to store the virtual machine on and select the required disk format and network mapping;
Review the summary information and select Finish;
The VMware Management Appliance will be provisioned in several minutes.

When the vMA is deployed, boot it and configure it. If you need help, check the vSphere 5.1 Documentation Center.

Important! Excluded the vMA from VMware DRS to make sure it is always hosted on the same server (last server in the server list, see below)

The next step is copying the PCNS files which you downloaded from the APC website to the vMA. I used ‘/tmp/APC’, to unzip the files use:

cd /opt/temp sudo gunzip pcns301ESXi.tar.gz sudo tar –xf pcns301ESXi.tar

There will now be a new directory named ESXi created in /tmp/APC, go to this directory and change the permissions of the ‘install_en.sh’ file so you can execute it and run the installer.

sudo chmod 777 install_en.sh sudo ./install_en.sh

Follow the instructions, accept the license, select a installation path, add a target ESXi host, etc. When the installation is complete you can configure PCNS from your web browser.

In the PCNS Configuration wizard provide:

Username and passwords to communicate with the APC’ Network Management Card (NMC) and to log in to PCNS;
IP address to communicatie with the NMC;
the UPS configuration, Single, Redundant, or Parallel; Single: In a Single-UPS configuration, each computer server or group of servers is protected by a single UPS. That is, each server has one PCNS agent communicating with a single NMC installed on a UPS. Redundant: In a Redundant-UPS Configuration, PowerChute Network Shutdown recognizes a group of either two or three UPS’s as a single UPS. In this configuration, one PowerChute Network Shutdown Agent on a server communicates with two or three NMCs (depending on the number of UPS’s in the configuration). Typically, the servers have multiple (dual or triple) power cords. Each UPS has its own NMC, which has a unique IP address. Parallel: In a Parallel-UPS Configuration, two or more UPS’s support the load. Each UPS does not need to be capable of supporting the load on its own as the combined output of all UPS’s in the configuration share the load. However, depending on the size of the load there may be reserve UPS(s) available. Redundancy is provided if there is at least one more UPS than is required to support the load.
UPS detail like MNC protocol, port and IP address;

When done correctly the PCNS will register with the NMC.

The next step is to configure how PCNS responds to UPS events. For instance, configure to shutdown the ESXi host when the UPS capacity drops below 50%. Beware: make sure that the remaining capacity gives you enough time to shutdown all virtual machines and ESXi hosts.

When that’s done, configure what actions PCNS needs to perform when initiating a shutdown, notify users, run scripts, etc.

Now you’re done with the PCNS configuration. The next step left is to add all VMware ESXi hosts in your cluster to the server list to allow for the shutdown of multiple host by a single vMA. To do this run the following command:

sudo vifp addserver 10.218.45.20

Verify the server has been added

vifp listservers

Important! The address of the host running the vMA should always be last in the list.

Next, add the server to the fasspass list with the command:

vifptarget –s

To verify correct communications between the client and the host run the command:

Vicfg-nics –l

Now we are almost done, the last step is to configure the virtual machines to shutdown with the host server and configure the ESXi shutdown order. In the vSphere client or web client select the ESXi host configuration.

Select the Virtual Machine Startup/Shutdown link. Click properties at the top right and use “Move Up” to move the VMs in the order you want leaving the vMA VM at the top. Note that the startup order is the only one displayed. Shutdown uses the startup list in reverse. If you want the VMs to start on their own move them all the way up to Automatic Startup still leaving the vMA VM at the top.

Next make sure that all virtual machines have the VMWare Tools installed for a graceful shutdown to occur. Check the virtual machine configuration to make sure the settings for shutdown are correct. Right click the VM -> Edit Settings -> Options Tab -> VMWareTools. Under Power Controls Stop should be set to “Shut Down Guest”.

This will gracefully shutdown the VMs in the order you set before telling the ESXi host to power down. You should see the shutdown commands in the log start when connected to the ESXi host with the vSphere client. If the VM fails to gracefully shutdown check to make sure VMWare Tools is running on the VM under its Summary tab in vCenter.

Now you’re done and ready to enjoy your worry-free weekends.

Option 2 (other UPS):

I also found a more generic solution from Ralph Lueders. Ralph wrote a very nice script to shutdown VMware ESXi server simply by checking the state of a specific ESXi network interface. To make this work you will need to connect the selected interface to a switch which is not protected by a UPS. If the specific interface is down for a number of cycles, the VMware ESXi and the running virtual machines are shutdown.

First of all download the auto-shutdown.sh script (change the extension after download).

When running the script you can select the monitored interfaces (-i) and the the number of down cycles (-t). Try auto-shutdown.sh -v for further information.

To start monitoring on ESXi 4 hosts, add the following to ‘/etc/rc.local’.

/bin/kill $(cat /var/run/crond.pid) /bin/echo “* * * * * /vmfs/volumes/datastore/control/auto-shutdown.sh -t 15 -i vmnic2” » /var/spool/cron/crontabs/root /bin/busybox crond

To start monitoring on ESXi 5 hosts, add the following to ‘/etc/rc.local.d/local.sh’ (before the exit 0)

/bin/kill $(cat /var/run/crond.pid) /bin/echo “* * * * * /vmfs/volumes/datastore/control/auto-shutdown.sh -t 15 -i vmnic2” » /var/spool/cron/crontabs/root /bin/crond

This will check the state of vmnic2 every minute and if vmnic2 is down for 15 minutes, the ESXi host will shut down. You will have to apply this on every ESXi host connected to the UPS.

Don’t forget to execute ‘auto-backup.sh’ to save the changes.

We are almost done, like with the APC UPS, the last step is to configure the virtual machines to shutdown with the host server and configure the ESXi shutdown order. In the vSphere client or web client select the ESXi host configuration.

Now you’re done and ready to enjoy your worry-free weekends.