Google Releases Open-Source Toolkit for Cloud Provider Benchmarking

New Google Perfkit Benchmarker has the ability to compare performance values across Google Engine, Amazon Web Services, and Microsoft Azure cloud platforms.

Comparing Public Cloud Platforms isn’t always easy—especially when it comes to performance evaluation. Price and feature chart comparison can only get you so far. Last month Google made that task a lot easier by releasing their new Perfkit Benchmarker. Perfkit is, in Google’s own words, a means to “…define a canonical set of benchmarks to measure and compare cloud offering.” Shorthand: Perfkit compares performance across public cloud IaaS providers. So far, the Perfkit tools support Google’s Compute Engine, Microsoft’s Azure, and Amazon’s AWS.

Sample Perfkit Explorer Dashboard

Perfkit runs about 20 different benchmark tests after installation, and has the unique ability to measure the end to end time to provision resources in cloud, while also providing standard peak performance metrics. Google’s team also released a Perfkit Explorer tool to make interpretation and comparison of results easier for users. Perfkit Explorer comes with a set of pre-built dashboards and gives users immediate access to data from network performance internal tests.

Over the last year, Google has worked on this project with more than 30 different researchers, customers and companies, including competitors such as Microsoft, CenturyLink, and Rackspace. Perfkit’s source code has been released under an ASLv2 license, which Google says “[makes] it easy for contributors to collaborate and maintain a balanced set of benchmarks.”

Serverbear.com

There are other Benchmarking tools available, such as Serverbear.com, which compares web and cloud servers, CloudHarmony’s Cloud Scores, who offer cloud-performance updates, and Cloud Spectator, who contributed to Google’s Perfkit project, but to my knowledge none have offered an open-source service with the same versatility and scale as Perfkit.

We’re excited to see Google releasing an open-source tool that allows users to pit Google’s Compute engine directly against its two biggest competitors, and we’re hoping that Perfkit will be expanded to support comparisons to other Cloud providers soon. Google describes Perfkit as a “living benchmark framework, designed to evolve as cloud technology changes.” We’ll be watching closely to see if it will live up to that statement as the toolkit gains popularity.