BenchmarkXPRT Development Community Blog

More details to come

on July 10, 2014

As we’ve been saying the past couple of months, we’re working on a benchmark for Chrome OS. The experimentation phase is winding down, and we are starting to shape the code into a useable benchmark. The design plan will leverage existing WebXPRT tests, of course. However, we’ve gone far beyond that. The benchmark will include video playback, 3D modeling via WebGL, and even an HTML5 game. The test also uses Chrome OS’ native execution capability. The benchmark will actually use the Portable Native Client (PNaCl), as PNaCl is the recommended tool chain for native client. It also gives the benchmark the ability to run on more platforms.

As we mentioned before, we’re including a battery test as part of the new benchmark. So far, we haven’t found a way to remove the requirement to put the device in developer mode for the battery test.

Next week, we’ll publish a design document for the community to review. As always, the design document is based on the comments and suggestions we received combined with our own research and experimentation.

Eric

Comment on this post in the forums

Posted in Chrome OS, Future of performance evaluation, HTML5, Performance benchmarking, WebXPRT |

It makes a difference

By Eric Hale

on July 3, 2014

Ars Technica reported this week that they tested the developer preview of Android L and saw a whopping 36 percent improvement in battery life! Google made improving battery life a priority, and it sounds like they are succeeding. I can’t wait to test Android L with BatteryXPRT.

This is a spectacular example of how a change in software can change benchmark results, but it’s hardly unique. I’ve written before about how background activity on a phone depressed my friend’s WebXPRT scores. AnandTech used both IE 11 and Chrome 30 to test the Surface Pro 2 with a variety of benchmarks, including WebXPRT, SunSpider, Octane, Browsermark, and others. Browser choice had a noticeable impact on results – about a 40 percent difference for WebXPRT and a 76 percent difference for SunSpider!

People are generally pretty aware that changing the hardware changes performance. However, sometimes they lose track of software differences. When you compare scores, it’s not always possible to keep all the variables the same, but it’s crucial to know what the differences are.

In other BenchmarkXPRT news, we’re making some final adjustments to HDXPRT 2014, and the general release is just around the corner.

Eric

Comment on this post in the forums

Posted in Android, Battery life, HDXPRT 2014, Performance benchmarking, Performance testing on tablets, WebXPRT |

The updated BatteryXPRT is here

By Eric Hale

on June 30, 2014

As we mentioned a couple of weeks ago, we discovered stability issues when running BatteryXPRT 2014 for Android on devices with only 1 GB of RAM. We now have a build that’s stable on those devices. Battery life and performance results from the new build are equivalent to those from the previous build, so your existing results are still valid and comparable.

You can download the new build of BatteryXPRT 2014 for Android at BatteryXPRT.com or through the Google Play Store. If you have any questions, please don’t hesitate to ask.

Eric

Comment on this post in the forums

Posted in Android, BatteryXPRT 2014 for Android |

In the works

By Eric Hale

on June 26, 2014

A couple of weeks ago, we mentioned stability issues with BatteryXPRT 2014 for Android when running on systems with only 1 GB of RAM. After some restructuring, we now have a build that’s stable on those devices. We’re finishing testing, and it’s looking good. We’ll be making the new build available at BatteryXPRT.com and in the Google Play store early next week. Battery life and performance results from the new build are equivalent to those from the previous build, so your existing results are still valid and comparable.

Speaking of BatteryXPRT, we have something special coming soon. I can’t say what it is yet, but we haven’t done anything like it before.

We’re also getting closer to a general release for HDXPRT 2014. As we’ve explained before, the new HDXPRT runs three iterations in less than 2 hours, and will be downloadable, so the entire installation and testing process is much quicker than before. We’re currently working out a few kinks, but we expect the general release very soon.

Also, we’re still accepting ideas for naming the Chrome-related XPRT now in development. Share your ideas at BenchmarkXPRTsupport@principledtechnologies.com.

Eric

Comment on this post in the forums

Posted in BatteryXPRT 2014 for Android, Chrome OS, Collaborative benchmark development, HDXPRT 2014 |

Testing the waters

By Eric Hale

on June 19, 2014

A couple of weeks ago, we talked about some of our ideas for a new XPRT designed for Google’s Chrome OS. We’ve been working with some of these ideas and, while we’re still in the experimental stage, things look promising so far.

As we mentioned in the earlier blog, we’re trying WebXPRT as a base for the performance part of the test. So far, the performance component is working well. In addition to modified WebXPRT tests, we’re also trying some things that are not part of the WebXPRT 2013 workload.

We’ve been able to get battery life, but it’s been challenging and we haven’t found a way to avoid using Chrome’s Developer Mode. Accessing Developer Mode in Chrome can be tricky and requires different steps for each hardware manufacturer. We’re hoping to find ways to make battery life testing easier.

I’ve been vague about the tests because they’re likely to change over the next few weeks. We’re experimenting with both browser-based and Native Client-based performance tasks. As they firm up, I’ll be able to share more information.

Challenges aside, we’re excited about this new benchmark, and committed to making it as effective as possible. We’d still love feedback on a name, so feel free to contact us at BenchmarkXPRTsupport@principledtechnologies.com with your ideas.

Eric

Comment on this post in the forums

Posted in Battery life, Chrome OS, Collaborative benchmark development, Performance benchmarking, What makes a good benchmark? |

A new version of BatteryXPRT is coming

By admin

on June 12, 2014

We’ve found that BatteryXPRT 2014 for Android has some stability issues on some devices with low memory. We’ve investigated the problem and found that the problem is that the low RAM reduces the per-application heap quota so much that BatteryXPRT may not run correctly.

To address this problem, we’ve restructured BatteryXPRT so that it stays within the heap limits on devices with 1 GB of RAM. We are currently testing the restructured version of BatteryXPRT and it looks good so far. It takes a while to thoroughly test BatteryXPRT, but we will release the new build as soon as we complete testing it.

We haven’t seen any change in the performance or battery life results in the testing so far. You’ll be able to safely compare your existing results with the results from the rebuilt version.

Remember that we are looking for ideas, and a really great name, for a Chrome OS benchmark. If you have any ideas, or are curious about the changes to BatteryXPRT , let us know!

Eric

Posted in Android, BatteryXPRT 2014 for Android |

More details to come

It makes a difference

The updated BatteryXPRT is here

In the works

Testing the waters

A new version of BatteryXPRT is coming

Check out the other XPRTs: