We believe that one of the most important things we can do is make the source code available. We believe that increasing the transparency of the benchmarking process and stimulating the participation of industry players and the public in the definition, development, understanding, and auditing of the benchmarks will lead to better benchmarks.
You may be thinking “Then why not open source the benchmark?” The short answer is that we need to make sure that the results from any version of our benchmarks are ones you can trust. You can watch Bill discuss this in BenchmarkXPRT: It’s not a benchmark.
We believe that the community model—which gives you total visibility into the benchmark, allows you to run your own experiments and contribute to future versions of the benchmark, and still protects the integrity of the results—strikes the right balance.
If you’re not a member, please consider joining. It’s easy.
If you are a member, check out the code and tell us how it can be better!
As we said in It’s finally here!, the Android ecosystem is very diverse and we knew that MobileXPRT might have problems on some devices. So far, we have encountered a couple of issues:
The Slideshow test failed during post validation check on the Onda tablet.
The Zoom-n-pinch test was unstable, failing part of the time, on the Zopo phone
On Friday, we will be releasing MobileXPRT 2013 community preview 1.1 (CP1.1), which fixes these problems. The results from version CP1.1 are comparable to the current community preview. If you have any problems, or questions, please e-mail us at firstname.lastname@example.org
We will release the source for CP1.1 next week.
In other news, we released a new video this week, BenchmarkXPRT: It’s not a benchmark. In this video, Bill answers some common questions about the BenchmarkXPRT Development Community. If you’ve ever wondered what exactly BenchmarkXPRT is, or why the world needs more benchmarks, this is the video for you.
Last week was possibly our biggest week ever. We announced PhoneXPRT, a new benchmark for evaluating the performance of smartphones, and released TouchXPRT 2013 and WebXPRT 2013 to the general public.
Since then, there’s been a lot of interest. The numbers keep going up! We’re not just talking about page views – people are downloading TouchXPRT and lots of people are running WebXPRT.
People have also been downloading the TouchXPRT source, which is very exciting. We strongly encourage community members to look at how the benchmark is put together. If you have programming skills and want to submit code, get in touch with us at email@example.com. We’d love to hear from you!
Oh, and there’s a new video! This one introduces WebXPRT to the public. It gives a good idea of the range of devices WebXPRT will run on.
That’s a lot for one week. We’re not resting on our laurels, though. Obviously, we’re working on PhoneXPRT. However, let’s not forget about HDXPRT. The comment period for HDXPRT 2013 officially closed on March 6 and we are starting to work on the HDXPRT 2013 design document. If you have any feedback you haven’t sent, please do send it on. We’ll do our best to incorporate it into the design document.