Category: Collaborative benchmark development

It’s almost here!

on August 13, 2015

This Friday, we will be releasing the community preview of MobileXPRT 2015. BenchmarkXPRT Development Community members will be able to download the installation APK from the MobileXPRT tab in the Members’ Area. We are happy to say that, although we thought that MobileXPRT 2015 would only run on Android 5.0 and above, we have been able to extend its compatibility back to Android 4.4 (KitKat).

As we’ve said, the new MobileXPRT is a 64-bit application. Although it will work on both 32-bit and 64-bit hardware, the code will automatically select 64-bit when available. If you want to compare 32-bit performance to 64-bit on a single device, MobileXPRT 2013 will still be available. It’s not a problem to have both versions of MobileXPRT installed at the same time, and results from the two versions of the benchmark are comparable. On the same note, remember that the UX tests are no longer available in MobileXPRT 2015, but they are available as part of MobileXPRT 2013 if you need them.

As always, the results from the community preview will be comparable to the general release.

We hope you enjoy the new MobileXPRT, and we look forward to seeing your results!

Eric

Explaining the BenchmarkXPRT Development Community

By Bill Catchings

on August 6, 2015

Over the last year, I’ve spoken about the XPRT benchmarks with people across America, in China (at IDF Shenzhen), and in Europe (at Mobile World Congress). I regularly found myself having to explain how the BenchmarkXPRT Development Community works. While I was glad to do so, I found myself wishing that the many people I wasn’t able to talk with could also learn about how the community works.

To help make that happen, we’ve developed a simple and engaging video. My (admittedly prejudiced) opinion is that it does a great job of explaining how the community works in less than two minutes. That’s a lot faster than I was able to explain it to folks!

We hope you enjoy the video. And we hope you’ll pass it along to other folks who aren’t already part of the community so they can learn how it works and hopefully be persuaded to join us. Thanks!

Bill

Posted in Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, IDF, What makes a good benchmark? |

An update on MobileXPRT 2015

By Eric Hale

on July 23, 2015

As I mentioned last week, we’ve been testing MobileXPRT 2015.

We’ve rebuilt the current MobileXPRT as a 64-bit application. This means that MobileXPRT 2015 will only run on Android 5.0 and above. For this reason, we’ll make MobileXPRT 2013 available for testing older versions of the operating system. Because the workloads haven’t changed, you’ll be able to compare results from MobileXPRT 2013 to those from MobileXPRT 2015.

Another change to MobileXPRT 2015 will be the removal of the UX tests. While these tests were useful on lower-end devices when MobileXPRT 2013 came out, they don’t distinguish modern devices. However, should you need them, these tests will continue to be available as part of MobileXPRT 2013.

There are a few other features that we’ll be talking about over the next few weeks. We’ll be releasing the MobileXPRT 2015 community preview before Android M is released. We’re looking forward to see how Android M performs!

Eric

Posted in Android, Android L, Android M, Collaborative benchmark development, Community Preview, MobileXPRT 2015 |

Something old, something new

By Eric Hale

on June 25, 2015

Last week, we talked about porting TouchXPRT 2014 to be a Windows 10 universal app. This will let it run on devices running Windows 10 and those running Windows 10 mobile.

We won’t be retiring TouchXPRT 2014 when we release the Windows 10 universal app version. Windows 8 doesn’t support Windows 10 universal apps, but Windows 10 will be able to run Windows 8 applications. This means you’ll also be continue to be able to use TouchXPRT 2014 to test Windows 8 based systems, as well as to compare Windows 8 and Windows 10 performance.

The results from TouchXPRT 2014 and the universal app version of the benchmark won’t be compatible. Even though the test scenarios will be the same, the porting process means that we have to change the APIs the benchmark is using and rebuild the benchmark with different tools.

We’re currently debating changing the way we version the benchmarks. As the number of versions of each benchmark increases, it may make sense to move away from year-based versioning. This will obviously affect what we call the new Windows 10 version of TouchXPRT. If you have any thoughts on this, please let us know!

Eric

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, TouchXPRT, TouchXPRT 2014, TouchXPRT release cycle, Windows 10, Windows 8 |

Rolling with the changes

By Eric Hale

on June 4, 2015

While WebXPRT 2015 has been running fine on earlier beta versions of Windows 10, we have found a problem on some recent versions. Starting with build 10122, the Local Notes test hangs when using the Microsoft Edge browser. (Note: This browser still identifies itself as Spartan in the builds we have seen.) Chrome and Firefox on Windows 10 have successfully run WebXPRT 2015, so the problem appears to be restricted to Edge/Spartan.

Because WebXPRT ran successfully on earlier builds of Windows 10, we are hoping that upcoming builds will resolve the problem. However, we have been investigating the issue in case there is something we can address. The problem is that the encrypted strings that the test is trying to write to LocalStorage are not being written correctly. Non-encrypted strings are being written correctly.

As soon as the problem gets resolved, we’ll let you know.

In other news, we’ve been looking at Android M. There are a lot of interesting changes coming, such as the difference in the way that Android M manages app permissions. We’ve decided to delay releasing the design document for the next version of MobileXPRT so that we can make sure that the design deals with these changes appropriately.

Eric

Posted in Android, Benchmarking, Chrome OS, Collaborative benchmark development, MobileXPRT, WebXPRT, WebXPRT 2015, Windows 10 |

It’s not the same

By Eric Hale

on May 21, 2015

We sometimes get questions about comparing results from older versions of benchmarks to the current version. Unfortunately, it’s never safe to compare the results from different versions of benchmarks. This principle has been around much longer than the XPRTs. A major update will use different workloads and test data, and will probably be built with updated or different tools.

To avoid confusion, we rescale the results every time we release a new version of an existing benchmark. By making the results significantly different, we hope to reduce the likelihood that results from two different versions will get mixed together.

As an example, we scaled the results from WebXPRT 2015 to be significantly lower than those from WebXPRT 2013. Here are some scores from the published results for WebXPRT 2013 and WebXPRT 2015.

Please note that the results above are not necessarily from the same device configurations, and are meant only to illustrate the difference in results between the two versions of WebXPRT.

If you have any questions, please let us know.

Eric

Posted in BenchmarkXPRT, Collaborative benchmark development, WebXPRT, WebXPRT 2013 results, WebXPRT 2015, What makes a good benchmark? |

Category: Collaborative benchmark development

It’s almost here!

Explaining the BenchmarkXPRT Development Community

An update on MobileXPRT 2015

Something old, something new

Rolling with the changes

It’s not the same

Check out the other XPRTs: