So far cloudcomputing.info and virtualization.info reported about a number of startups that offer a benchmark platform to compare the performance of multiple Infrastructure-as-a-Service (IaaS) and Platform-as-a-Service (PaaS) public clouds, including CloudHarmony, CloudSleuth and Bench the Cloud.
Unfortunately, performance is not the only thing hard to compare in public cloud computing offerings. Comparing pricing is another big challenge, as Geva Perry at Thinking Out Cloud highlights.
He suggests that the problem depends on at least four issues:
- Cloud providers use non-standard, obfuscated terminology
- Cloud providers use wildly varying pricing scheme
- Not all cloud offerings are created equal
- Cloud computing pricing is fluctuating rapidly
While the first and the last issues may go away over time, the second and third ones may not, as it would require a sort of standardization as we have today in the hosting industry.
The most interesting point anyway is his suggestion to compare the public clouds in the immediate:
There seems to be no escape from creating a test application and running it on multiple services to see where the cost comes out. Then again, that may turn out to be a very time-consuming and expensive effort that may not be worth it — at least not initially…
Quite the opposite, this may be a great idea, as an extension of a future benchmark platform for public clouds.
The real problem would be how to compare the tested application to the real application that a company may want to deploy off-premises.