VMGuru

About two weeks ago I attended a session at the VMware User Group meeting here in the Netherlands about Project VRC. After the presentation I asked myself: ‘What is the value of this project?’.

For you who don’t know what Project VRC is:

“Project Virtual Reality Check (VRC) is a joint venture of Log•in Consultants and PQR, who have researched the optimal configuration for the different available hypervisors (hardware virtualization layers). The project arises from the growing demand for a founded advice on how to virtualise Terminal Server and Virtual Desktop (VDI) workloads. Through a number of researches, Log•in Consultants and PQR show you the scaling possibilities for Terminal Server environments as well as Virtual Desktops.” http://www.virtualrealitycheck.net/

Don’t get me wrong: What they did was a very good initiative, it showed the performance differences between different hypervisors. Although the results were not that surprising it was good to see the validation numbers of the things we already knew.

I also think that the guys who did the project where totally surprised by the attention vendors and customers gave to the project. It was an outstanding (marketing) tool to show the value of virtualization and especially XenApp on a hypervisor. Because of this attention the whole project got out of hand. Although this was not the goal of the project, vendors and customers used it as a reference guide for vitalizing XenApp. That’s the point where I started to wonder what the real value of the project VRC was.

Not a real workload for the XenApp servers. The biggest problem with the tests in project VRC is, that they didn’t use real life workloads. Although it is quite difficult to simulate a real workload for a XenApp server with a tool, the “real” workload is not a real workload if you only use Office, Internet Explorer including Flash applets and Adobe Acrobat Reader. There is not a single customer I know that only uses this application set. They all have a lot more “business” applications, which have a lot more impact on the user experience and the user acceptance, than the set applications used in project VRC.

Used local disks The load was generated on local disks. A lot of customers I know don’t virtualize workloads on local disks; instead they use shared storage to store their virtual servers on. I think that the use of shared storage had made a huge difference in the results and would have approached more the real enterprise environments.

Maximize the number of users. Maximizing the number of users on itself is number to measure but the project used this numbers as absolute numbers to compare the different hypervisors. To me that is not relevant. I’m more interested in the numbers the different hypervisor gave on the best user experience in a 80% load rather than a 100% load. There is not a single enterprise that stresses the hardware to a 100%!

Wrong results. During phase 1 of the project the team forgot to make use of the processor capabilities to offload some of the most intensive virtualization instructions. Second, in the 1.0 publication the project team used a wrong measuring method which was affected bu clock drift. One of the vendors involved pointed out this wrong approach, Project VRC admitted this failure and conducted a new test and published a version 1.5 in which the wrong measuring method was corrected. After this correction the results for each of the hypervisors are pretty close to each other.

So what is the value of project VRC? It’s a nice initiative and it shows us the maximum number of sessions you can run with different hypervisors, but to me it is nothing more than that and it’s I good document for discussion. That’s the real value of project VRC.