BenchmarkXPRT Development Community Blog

Rolling with the changes

on June 4, 2015

While WebXPRT 2015 has been running fine on earlier beta versions of Windows 10, we have found a problem on some recent versions. Starting with build 10122, the Local Notes test hangs when using the Microsoft Edge browser. (Note: This browser still identifies itself as Spartan in the builds we have seen.) Chrome and Firefox on Windows 10 have successfully run WebXPRT 2015, so the problem appears to be restricted to Edge/Spartan.

Because WebXPRT ran successfully on earlier builds of Windows 10, we are hoping that upcoming builds will resolve the problem. However, we have been investigating the issue in case there is something we can address. The problem is that the encrypted strings that the test is trying to write to LocalStorage are not being written correctly. Non-encrypted strings are being written correctly.

As soon as the problem gets resolved, we’ll let you know.

In other news, we’ve been looking at Android M. There are a lot of interesting changes coming, such as the difference in the way that Android M manages app permissions. We’ve decided to delay releasing the design document for the next version of MobileXPRT so that we can make sure that the design deals with these changes appropriately.

Eric

Posted in Android, Benchmarking, Chrome OS, Collaborative benchmark development, MobileXPRT, WebXPRT, WebXPRT 2015, Windows 10 |

What makes a good benchmark?

By Eric Hale

on May 28, 2015

As we discussed recently, we’re working on the design document for the next version of MobileXPRT, and we’re really interested in any ideas you may have. However, we haven’t talked much about what makes for a good benchmark test.

The things we measure need to be quantifiable. A reviewer can talk about the realism of game play, or the innovative look of a game, and those are valid observations. However, it is difficult to convert those kinds of subjective impressions to numbers.

The things we measure must also be repeatable. For example, the response time for an online service may depend on the time of day, number of people using the service at the time, network load, and other factors that change over time. You can measure the responsiveness of such services, but doing so requires repeating the test enough times under enough different circumstances to get a representative sample.

The possible things we can measure go beyond the speed of the device to include things such as battery life and compatibility with standards, and even fidelity or quality such as with photos or video. BatteryXPRT and CrXPRT test battery life, while the HTML5 tests in WebXPRT are among those that test compatibility. We are currently looking into quality metrics for possible future tools.

I hope this has given you some ideas. If so, let us know!

Eric

Posted in Benchmarking, Benchmarks in general, BenchmarkXPRT development community, MobileXPRT |

It’s not the same

By Eric Hale

on May 21, 2015

We sometimes get questions about comparing results from older versions of benchmarks to the current version. Unfortunately, it’s never safe to compare the results from different versions of benchmarks. This principle has been around much longer than the XPRTs. A major update will use different workloads and test data, and will probably be built with updated or different tools.

To avoid confusion, we rescale the results every time we release a new version of an existing benchmark. By making the results significantly different, we hope to reduce the likelihood that results from two different versions will get mixed together.

As an example, we scaled the results from WebXPRT 2015 to be significantly lower than those from WebXPRT 2013. Here are some scores from the published results for WebXPRT 2013 and WebXPRT 2015.

Please note that the results above are not necessarily from the same device configurations, and are meant only to illustrate the difference in results between the two versions of WebXPRT.

If you have any questions, please let us know.

Eric

Posted in BenchmarkXPRT, Collaborative benchmark development, WebXPRT, WebXPRT 2013 results, WebXPRT 2015, What makes a good benchmark? |

WebXPRT 2015 source code is now available

By Eric Hale

on May 19, 2015

As of today, we are making the WebXPRT 2015 source code available to community members.

Download the WebXPRT 2015 source here (login required).

We’ll also post a link to the source on the WebXPRT tab in the Members Area. The source code package contains instructions for setting up a local installation of WebXPRT. However, please note that you must test using WebXPRT.com to get an official result.

If you want more information, please contact BenchmarkXPRTSupport@principledtechnologies.com.

We look forward to your feedback!

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Source code, WebXPRT 2015 |

What are the implications?

By Eric Hale

on May 14, 2015

It’s been a couple of weeks since the Microsoft Build 2015 conference. There was a lot of interesting news and we are still digesting what it means for the XPRTS, especially TouchXPRT.

Rebuilding TouchXPRT as a universal app has the potential to let it run on a much wider range of devices: PCs, tablets, phones, even the Xbox. This would give TouchXPRT the kind of versatility that we enjoy in the Android space with MobileXPRT and BatteryXPRT.

It’s a lot more complicated to sort out the implications of Microsoft Continuum, which allows you to use your phone as a computer by connecting it to a docking station. The features of your device and the way the apps behave can change based on the display available. Connect the phone to a docking station and it behaves like a desktop. It also means that the hardware and features available on a device could potentially change while you are testing the device. TouchXPRT would need to detect any such changes and respond appropriately.

That’s a lot to think about, and we’ve been experimenting. If you have any thoughts about Windows 10 and benchmarking, please let us know.

Eric

Posted in BatteryXPRT 2014 for Android, MobileXPRT, TouchXPRT, TouchXPRT release cycle, Windows 10 |

WebXPRT 2015 is here!

By Eric Hale

on May 8, 2015

Today, we’re releasing WebXPRT 2015, our benchmark for evaluating the performance of Web-enabled devices. The BenchmarkXPRT Development Community has been using a community preview for several weeks, but now that we’ve released the benchmark, anyone can run WebXPRT and publish results.

Run WebXPRT 2015

WebXPRT 2013 is still available here while people transition to WebXPRT 2015. We will provide plenty of notice before discontinuing WebXPRT 2013.

After trying out WebXPRT, please send your comments to BenchmarkXPRTsupport@principledtechnologies.com.

Posted in Automation, BenchmarkXPRT development community, Performance benchmarking, Performance of computing devices, Performance testing on tablets, WebXPRT, WebXPRT 2015 |

Rolling with the changes

What makes a good benchmark?

It’s not the same

WebXPRT 2015 source code is now available

What are the implications?

WebXPRT 2015 is here!

Check out the other XPRTs: