Speedometer 2.1, a tool for benchmarking web browsers across various workloads, is still available from Apple. To alleviate user concerns about possible bias, the software’s next major iteration will include input from three browser vendors. The new project is still in its early stages, but the collaboration represents a watershed moment.
Apple, Google, and Mozilla have announced that they will collaborate on the next version of the Speedometer web browser benchmark in order to better represent modern browser tasks. The new version is still in beta, but users can begin testing it right away.
Speedometer 3 is designed to test the full range of how normal users operate browsers by running on normal web pages instead of benchmarking in a tight environment. The developers plan to continually tweak the software based on up-to-date data on what features are most important to everyday users. Google confirmed the benchmark will represent JavaScript frameworks and other modern workloads.
It will be a while before Speedometer 3 is fully up and running, but more information about it will emerge in the coming months. If the collaboration works out, it should give web designers a better picture of how their work will run across multiple browsers.
We’re glad to be collaborating with @WebKit and @GoogleChrome on Speedometer 3. 🌪️
Here’s why this work is important for the Web 🧵
— Mozilla Developer 👩🏾💻 (@mozhacks) December 15, 2022
Currently, most users who want to see how fast their browser responds to web applications should probably use Speedometer 2.1. However, because Apple operates 2.1, it’s easy to assume its results could be biased against Safari’s competitors like Google Chrome, Mozilla Firefox, or Microsoft Edge. A benchmark based on WebKit might not fully understand the performance challenges of Blink, V8, or SpiderMonkey, for example.
Although Apple’s WebKit GitHub page hosts Speedometer 3, Google and Mozilla will develop it alongside the Cupertino giant. This should ensure it treats all three browsers fairly and that it accurately understands how users engage with them. Mozilla’s announcement noted that a good browser benchmark requires cooperation between browser vendors, website builders, framework builders, and web standards groups.
A governance policy based on consent between the three browser giants will guide Speedometer 3’s development. Trivial changes require just one of the companies’ approval, non-trivial changes need two to agree on them, and significant changes necessitate consensus between all three.