Category: Future of performance evaluation

Time for a merger of equals?

on April 16, 2015

As I said last week, we’re working on the design document for the new version of MobileXPRT, and we expect to have it out in the next couple of weeks. We have several ideas we’re pretty excited about.

One of the ideas we’ve been considering is merging BatteryXPRT and MobileXPRT into a single benchmark. This would be similar to what we’ve done with CrXPRT, which has tests for both battery life and performance. As with CrXPRT, you’d be able to run either test, and you could get performance and battery life for a device in a single day using a single benchmark.

If we as the community do decide to merge the benchmarks, there will be a lot to think about. For example, MobileXPRT is unaware whether it’s connected to the Internet, while BatteryXPRT not only detects how it’s connected, but selects the appropriate Airplane mode or Network-Wifi Mode test. And, of course, we’d have to figure out what to call it.

What do you think about merging the two benchmarks? Would it make your life simpler? What other features would you like to see in the new MobileXPRT? This is the time to speak up!

Eric

Posted in Battery life, BatteryXPRT 2014 for Android, BenchmarkXPRT development community, Collaborative benchmark development, CrXPRT, Future of performance evaluation, MobileXPRT |

More power, more control

By Eric Hale

on March 19, 2015

As I said last week, the community preview for WebXPRT 2015 is coming up soon. One of the changes that will be exciting to anyone who does a lot of testing is that we made it simpler to automate WebXPRT tests.

WebXPRT 2015 will let you automatically select any set of tests you want to run. However, as always, you must run the entire suite of tests to get an overall score. Although the community preview will not include any experimental tests, the automation includes control for those future tests as well.

You may choose from several output formats: HTML table, XML, and CSV, or you can download the results as a text file.

Using the automation is simple: you just append the desired test parameters to the end of the URL. The format allows you to mix and match a lot of options, while still being very concise. The details will be in the release notes.

As people who test a lot of devices, we are very excited about this new capability.

Eric

Posted in Automation, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, Performance benchmarking, WebXPRT, What makes a good benchmark? |

More details to come

By Eric Hale

on July 10, 2014

As we’ve been saying the past couple of months, we’re working on a benchmark for Chrome OS. The experimentation phase is winding down, and we are starting to shape the code into a useable benchmark. The design plan will leverage existing WebXPRT tests, of course. However, we’ve gone far beyond that. The benchmark will include video playback, 3D modeling via WebGL, and even an HTML5 game. The test also uses Chrome OS’ native execution capability. The benchmark will actually use the Portable Native Client (PNaCl), as PNaCl is the recommended tool chain for native client. It also gives the benchmark the ability to run on more platforms.

As we mentioned before, we’re including a battery test as part of the new benchmark. So far, we haven’t found a way to remove the requirement to put the device in developer mode for the battery test.

Next week, we’ll publish a design document for the community to review. As always, the design document is based on the comments and suggestions we received combined with our own research and experimentation.

Eric

Comment on this post in the forums

Posted in Chrome OS, Future of performance evaluation, HTML5, Performance benchmarking, WebXPRT |

The name game

By Eric Hale

on June 5, 2014

In Something shiny, we discussed the leading contender in our search for new benchmark ideas, a benchmark tailored especially for the Chrome OS, and we’ve been looking at what workloads would make sense.

As we said, the ability to measure battery life would be useful. That’s not easy in the Chrome environment. We think we may be able to do it, but the Chromebook may have to be in developer mode. Even so, we can leverage what we’ve learned from BatteryXPRT to get a reliable estimate of battery life in less than a working day.

Measuring performance, however, is a must. We’ve been looking at the existing WebXPRT workloads as well as other applications, such as education apps, online games, HD video playback, music playback, and more. We’re also looking for areas where using native client execution makes sense, such as higher-resolution photo editing.

In addition, we’re thinking about what we might call this benchmark. ChromeXPRT would be obvious, but probably wouldn’t pass Google’s naming restrictions.

Do you have ideas for the benchmark’s name? Are there Chrome-based benchmark workloads you’d love to see? Let us know at BenchmarkXPRTsupport@principledtechnologies.com!

Eric

Comment on this post in the forums

Posted in Chrome OS, Collaborative benchmark development, Future of performance evaluation, WebXPRT |

Stronger, faster, and definitely better

By Eric Hale

on May 29, 2014

This week we released HDXPRT 2014 Community Preview 1 (CP1) to BenchmarkXPRT Development Community members. For those community members who’ve been around since the beginning, you’ll notice how much HDXPRT has changed. We’ve trimmed down HDXPRT without reducing the value of the test. While HDXPRT 2012 required multiple installation DVDs, HDXPRT 2014 CP1 is available for download in the member’s area. In addition, it took HDXPRT 2012 at least five hours to complete the recommended three test iterations. HDXPRT 2014 is much faster – the total time required for install and three test iterations is less than two hours!

Like all versions of HDXPRT, CP1 uses real-world media applications such as Adobe® Photoshop® Elements 12, Apple® iTunes®11.1, CyberLink MediaEspresso 6.7, and others to perform common consumer tasks like editing photos, converting videos, and editing music files.

If you’re a community member, give CP1 a try. If you have questions about HDXPRT 2014 or any of the XPRTs, please contact us at BenchmarkXPRTsupport@principledtechnologies.com. If you’d like to be a part of future XPRT development and community previews, now’s a great time to join!

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, HDXPRT 2012, HDXPRT 2014 |

Something shiny

By Eric Hale

on May 22, 2014

As I mentioned in Looking for the next big thing, we’ve received some ideas for benchmarks recently. While there were several good ideas, the one that rose to the top of the list was a benchmark for Chrome-based devices.

For the past year or so, Chromebooks have been the fastest-growing segment of the PC market, so there’s a lot of interest. Given the rising profile of Chromebooks, a Chrome OS benchmark seems like a good fit with the existing XPRTs.

We’ve just started batting around ideas about what a Chrome OS benchmark might look like. Because the browser is the interface, WebXPRT is the natural place to start. However, WebXPRT runs on all platforms. We would like a Chrome benchmark to consider Chrome‑specific technologies like Google Native Client. If possible, it would also be great to get the battery life as well as performance.

Those are a few of our ideas. Do you have ideas about what you would like to see in a Chrome OS benchmark? If so, what would you consider most important for it to include? Let us know at benchmarkxprtsupport@principledtechnologies.com!

And remember, we’re always looking for new ideas. If there’s a benchmark you’d love to see, let us know.

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, Future of performance evaluation, Google Chrome, WebXPRT, What makes a good benchmark? |

Category: Future of performance evaluation

Time for a merger of equals?

More power, more control

More details to come

The name game

Stronger, faster, and definitely better

Something shiny

Check out the other XPRTs: