BenchmarkXPRT Development Community Blog

We haven’t mentioned this in a while

on October 15, 2015

I had a conversation with a community member yesterday who wanted to know whether we would test his device with one of the XPRTs. The short answer is “Absolutely!” The somewhat longer answer follows.

If you send us a device you want us to test, we will do so, with the appropriate set of XPRTs, free of charge. You will know that an impartial, third-party lab has tested your device using the best benchmarking practices. After we share the results with you, you will have three options: (1) to keep the results private, (2) to have us make the results public immediately in the appropriate XPRT results databases, or (3) to delay releasing the results until a future date. Regardless of your choice, we will keep the device so that we can use it as part of our testbed for developing and testing future versions of the XPRTs.

When we add the results to our online databases, we will cite Principled Technologies as the source, indicating that we stand behind the results.

The free testing includes no collateral beyond publishing the results. If you would like to publicize them through a report, an infographic, or any of the other materials PT can provide, just let us know and the appropriate person will contact you to discuss the how much those services would cost.

If you’re interested in getting your device tested for free, contact us at BenchmarkXPRTSupport@principledtechnologies.com.

Eric

Posted in Benchmarking, Benchmarking computing devices, BenchmarkXPRT, BenchmarkXPRT development community, Principled Technologies |

An update on TouchXPRT 2016

By Eric Hale

on October 8, 2015

We’ll be releasing the MobileXPRT 2015 white paper tomorrow. It contains lots of information about MobileXPRT 2015 that you won’t find anywhere else. We hope you’ll find it very informative.

A couple of weeks ago, we released the design document for TouchXPRT 2016 (login required). This week, we put the first build of TouchXPRT 2016 into testing. It’s a Universal Windows app that runs on Windows 10 tablets, PCs, and phones. This means that TouchXPRT can now run on a wider variety of devices. However, it also means that TouchXPRT 2016 will not be backward compatible with Windows 8 and 8.1.

Given the current state of the SDKs, installing the test builds on phones is more complicated than we would like. We’re looking into ways to simplify the install before releasing the community preview. Testing on phones is particularly important because we made many of the UI changes to enable TouchXPRT to work acceptably on a small display.

We’ll keep you informed as testing proceeds. We’re hoping to release the community preview in the next couple of weeks.

Eric

Posted in Android, MobileXPRT, MobileXPRT 2015, TouchXPRT, TouchXPRT 2016, TouchXPRT release cycle, White papers, Windows 10, Windows 8, Windows 8.1 |

Question we get a lot

By Eric Hale

on October 2, 2015

“How come your benchmark ranks devices differently than [insert other benchmark here]?” It’s a fair question, and the reason is that each benchmark has its own emphasis and tests different things. When you think about it, it would be unusual if all benchmarks did agree.

To illustrate the phenomenon, consider this excerpt from a recent browser shootout in VentureBeat:

While this looks very confusing, the simple explanation is that the different benchmarks are testing different things. To begin with, SunSpider, Octane, JetStream, PeaceKeeper, and Kraken all measure JavaScript performance. Oort Online measures WebGL performance. WebXPRT measures both JavaScript and HTML 5 performance. HTML5Test measures HTML5 compliance.

Even with benchmarks that test the same aspect of browser performance, the tests differ. Kraken and SunSpider both test the speed of JavaScript math, string, and graphics operations in isolation, but run different sets of tests to do so. PeaceKeeper profiles the JavaScript from sites such as YouTube and FaceBook.

WebXPRT, like the other XPRTs, uses scenarios that model the types of work people do with their devices.

It’s no surprise that the order changes depending on which aspect of the Web experience you emphasize, in much the same way that the most fuel-efficient cars might not be the ones with the best acceleration.

This is a bigger topic than we can deal with in a single blog post, and we’ll examine it more in the future.

Eric

Posted in Benchmark metrics, Benchmarking, Benchmarking computing devices, BenchmarkXPRT development community, Collaborative benchmark development, Performance benchmarking, Performance of computing devices, WebXPRT, What makes a good benchmark? |

MobileXPRT 2015 source code and TouchXPRT 2016 Design Overview are available

By Eric Hale

on September 25, 2015

We’re excited to announce that the MobileXPRT 2015 source code and TouchXPRT 2016 Design Overview are now available to community members!

Download the MobileXPRT 2015 source here (login required).

Download the MobileXPRT 2015 build instructions here (login required).

Download the TouchXPRT 2016 Design Overview here (login required).

We also posted links to all three items on the MobileXPRT and TouchXPRT tabs in the Members’ Area.

If you want more information, please contact BenchmarkXPRTsupport@principledtechnologies.com.

We look forward to your feedback!

Posted in BenchmarkXPRT, BenchmarkXPRT development community, MobileXPRT, MobileXPRT 2015, Source code, TouchXPRT, TouchXPRT 2016, TouchXPRT development process |

The benefits of membership

By Eric Hale

on September 24, 2015

We have a couple of goodies for community members coming tomorrow.

The TouchXPRT 2016 design overview will tell you what we’re planning for the upcoming community preview. Thanks to everyone who’s contributed ideas. Let us know if the design overview omits anything you’d like to see in the benchmark.

The MobileXPRT 2015 source code and the instructions for building MobileXPRT 2015 will be available as well. Community members have access to the source for all the XPRT benchmarks. Making the source available is a pillar of the community model.

Look for the design overview and source code in the members’ area.

If you aren’t yet a member, this is a great time to join!

Eric

Posted in Collaborative benchmark development, MobileXPRT, MobileXPRT 2015, TouchXPRT, TouchXPRT 2016, TouchXPRT development process, TouchXPRT release cycle |

Impressed and Excited

By Eric Hale

on September 17, 2015

A couple of weeks ago we talked about some initiatives we’ve been exploring. This week, we’re happy to be able to talk about a new project. We’re sponsoring a senior project with the computer science department at North Carolina State University (NCSU).

As part of their education, small teams of seniors work with local companies on various programming projects. The students are expected to put significant time into these projects, and the tasks aren’t easy. You can get a sense of the range and complexity of these projects by looking at past project proposals.

We submitted a proposal for creating an experimental benchmark test to NCSU and they accepted it and assigned us a team of students. I’ve met a couple of times now with them. I’m impressed and very excited about what they’re going to do. We’re not ready to talk about the project yet, but if they’re successful, we’ll make the new test available on the BenchmarkXPRT site. We might even include it in a future XPRT!

Our hope is that if this project is successful, we can replicate it at other schools and help train the next generation of benchmark developers. As a bonus, the BenchmarkXPRT community will get some fresh perspectives and some new experimental test tools.

Eric

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation |

We haven’t mentioned this in a while

An update on TouchXPRT 2016

Question we get a lot

MobileXPRT 2015 source code and TouchXPRT 2016 Design Overview are available

The benefits of membership

Impressed and Excited

Check out the other XPRTs: