No internet connection results in slow vSphere client consoles
In the last few weeks a customer that I am working for has been making a lot of changes within their infrastructure. Some big and some (on the surface) small. Somewhere during those weeks a change was made and the consequence of that change has gone by unnoticed at first. Then reports started to come in from colleague administrators that console sessions for virtual machines, when using the vSphere client, where really slooooowwwww. Opening a console took more than 10 seconds and trying to open more simultaneous would freeze the users screen entirely.
As mentioned before there had been several changes in the environment, the biggest one being the migration to a new DNS appliance and implementing IPv6. So the first thought was that it would be related to DNS resolving. But after several connection tests we could find nothing wrong with resolving DNS records from the hosts or vCenter.
One of the goals of the customer is to migrate the vSphere infrastructure to vSphere 5.5 in the near future. And because we wanted to offer our colleagues time to prepare working with the vSphere webclient, the vSphere webclient plugin was installed on the management server (this is where all administrators work on). Thinking that the installation of the plug-in might had something to do with it I started a RDP session to the vCenter server and started the vSphere client that we had installed. Unfortunately this gave the same result, thus excluding the plug-in as the cause.
So what else could it be? We did tighten the firewall policies as there where some servers able to browse the internet, but what could that have to do with the vSphere client and console sessions. Turns out this actually can have an impact on the vSphere client. No internet connection can result in slow vSphere client consoles.
In this virtual infrastructure we are making use of self-signed SSL certificates instead of certificates issued by a CA. In response to this the vSphere client / Windows OS tries to update it’s own root certificates by connecting to www.download.windowsupdate.com. So after the new firewall policies where in place the server could not connect to this site anymore, but still was trying to do so everytime it needed to confirm a SSL certificate. And apparently opening a console session with a virtual machine is reason for checking the SSL certificate and eventually resulting in a “time-out”.
As we did not want to revert the firewall policy we decided to edit the local group policy on the server. The setting we changed is called “Turn off Automatic Root Certificates Update” and can be found in “Computer\Administrative Templates\System\Internet Communication Management\Internet Communication Setting”. Changing this setting to “enabled” prevents the server from updating it’s root certificates. The change to the setting takes effect immediately and the performance of the consoles returned to what they where before right away.
It took me a while before I found this solution and that is why I wanted to share it here. The article I used as background can be found here.