In the attempt to provide more transparency to those customers that are approaching cloud computing, a number of brave firms is building methodologies and infrastructures to benchmark the performance of several Infrastructure-as-a-Service (IaaS) and Platform-as-a-Service (PaaS) public clouds.
A little more than one month ago, CloudHarmony published its first performance analysis on the most popular IaaS public clouds.
This week Compuware goes beyond that, launching a public tracking service that measures the performance of IaaS and PaaS public clouds: CloudSleuth.
The CloudSleuth methodology is simple: Compuware deploys an identical target application to each cloud platform. The Gomez Performance Network (GPN) is used to run test transactions on the deployed target applications and monitor the response time and availability from various points around the globe. Hundreds of data points from each successive test run are collected and aggregated into a cloud performance database. The results are exposed online, for free, through the nifty Flash-based dashboard above.
What Compuware is trying to do here is rather odd: comparing the performance of different public clouds that are based on completely different architectures: while Amazon EC2 or The Rackspace Cloud are IaaS clouds, Microsoft Azure and Google App Engine (GAE) are PaaS clouds.
The company claims that its target application, a two web pages retail shopping site with 41 items, stay unchanged on all tested clouds, but it’s quite evident that everything below it is different, from the application server to the optional presence of a virtualization layer.
How’s the faster platform anyway? Right now the top three are:
- OpSource (IaaS)
- Google App Engine (PaaS)
- The Rackspace Cloud (IaaS)