Category: Future of performance evaluation

Another great year

on December 31, 2015

A lot of great stuff happened this year! In addition to releasing new versions of the benchmarks, videos, infographics, and white papers, we released our first-ever German UI and sponsored our first student partnership at North Carolina State University. We visited three continents to promote the XPRTs and saw XPRT results published in six of them (we’re still working on Antarctica).

Perhaps most exciting, we reached our fifth anniversary. Users have downloaded or run the XPRTs over 100,000 times.

As great as the year has been, we are sprinting into 2016. Though I can’t talk about them yet, there are some big pieces of news coming soon. Even sooner, I will be at CES next week. If you would like to talk about the XPRTs or the future of benchmarking, let me know and we’ll find a time to meet.

Whatever your holiday traditions are, I hope you are having a great holiday season. Here’s wishing you all the best in 2016!

Eric

Posted in Benchmarking, Benchmarks in general, BenchmarkXPRT development community, CES, Collaborative benchmark development, Future of performance evaluation, German, History of benchmarking |

Nebula Wolf

By Eric Hale

on December 10, 2015

A couple of months ago, we talked about the senior project we sponsored with North Carolina State University. We asked a small team of students to take a crack at implementing a game that we could use as the basis of a benchmark test.

Last Friday, the project culminated with a presentation at the annual Posters and Pies event.

The team gave a great presentation, and I was impressed by how much they accomplished in 3 months. They implemented a game that they called Nebula Wolf – the mascot for NC State is a wolf. It’s a space-themed rail shooter. You can play the game, or click a button to run a script for benchmarking purposes. In the scripted mode, Nebula Wolf unlocks the frame rate so the device can run at full speed.

Over the next couple of weeks, we’re going to be testing Nebula Wolf, digging into the code and getting a deeper understanding of what the team did. We’re hoping to make the game available on our web site soon.

Tomorrow, AJ, Brien, and Rachel will present one last time, here at PT. It’s been a real pleasure working with them. I wish them all good luck as they finish college and start their careers.

Eric

Posted in Benchmark metrics, Benchmarking, Collaborative benchmark development, Future of performance evaluation |

Impressed and Excited

By Eric Hale

on September 17, 2015

A couple of weeks ago we talked about some initiatives we’ve been exploring. This week, we’re happy to be able to talk about a new project. We’re sponsoring a senior project with the computer science department at North Carolina State University (NCSU).

As part of their education, small teams of seniors work with local companies on various programming projects. The students are expected to put significant time into these projects, and the tasks aren’t easy. You can get a sense of the range and complexity of these projects by looking at past project proposals.

We submitted a proposal for creating an experimental benchmark test to NCSU and they accepted it and assigned us a team of students. I’ve met a couple of times now with them. I’m impressed and very excited about what they’re going to do. We’re not ready to talk about the project yet, but if they’re successful, we’ll make the new test available on the BenchmarkXPRT site. We might even include it in a future XPRT!

Our hope is that if this project is successful, we can replicate it at other schools and help train the next generation of benchmark developers. As a bonus, the BenchmarkXPRT community will get some fresh perspectives and some new experimental test tools.

Eric

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation |

Upping our game

By Eric Hale

on August 27, 2015

As we wrote last week, we’re releasing MobileXPRT 2015 to the public tomorrow. Thanks to everyone who helped make the community preview a success!

We’re working on the TouchXPRT 2016 design document and will make it available for the community to review soon.

As you know, we’re always investigating initiatives that could improve our game. We’re continuing to investigate creating experimental tests for future XPRTs. Experimental tests will allow us to maintain broad compatibility for each XPRT tool while providing testers with an opportunity to evaluate cutting-edge technologies.

Another initiative involves looking for new partnerships with people who are not yet part of the community, but could add valuable input to the development process. It’s too soon to say much more about this, but we’re having fruitful conversations and hoping that these partnerships will grow the community even more!

If you have ideas about experimental tests, improving the XPRTs, or expanding the community, please let us know.

Eric

Posted in Android, Collaborative benchmark development, Community Preview, Future of performance evaluation, MobileXPRT, MobileXPRT 2015, TouchXPRT, TouchXPRT 2016, What makes a good benchmark? |

Something old, something new

By Eric Hale

on June 25, 2015

Last week, we talked about porting TouchXPRT 2014 to be a Windows 10 universal app. This will let it run on devices running Windows 10 and those running Windows 10 mobile.

We won’t be retiring TouchXPRT 2014 when we release the Windows 10 universal app version. Windows 8 doesn’t support Windows 10 universal apps, but Windows 10 will be able to run Windows 8 applications. This means you’ll also be continue to be able to use TouchXPRT 2014 to test Windows 8 based systems, as well as to compare Windows 8 and Windows 10 performance.

The results from TouchXPRT 2014 and the universal app version of the benchmark won’t be compatible. Even though the test scenarios will be the same, the porting process means that we have to change the APIs the benchmark is using and rebuild the benchmark with different tools.

We’re currently debating changing the way we version the benchmarks. As the number of versions of each benchmark increases, it may make sense to move away from year-based versioning. This will obviously affect what we call the new Windows 10 version of TouchXPRT. If you have any thoughts on this, please let us know!

Eric

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, TouchXPRT, TouchXPRT 2014, TouchXPRT release cycle, Windows 10, Windows 8 |

WebXPRT 2015

By Eric Hale

on May 7, 2015

Tomorrow we’ll be releasing WebXPRT 2015, with mirror site in Singapore to follow soon. We’ve been talking about it for a while and we’re delighted to finally make it available to the public.

As we’ve discussed over the past few weeks, the new WebXPRT is a big improvement over WebXPRT 2013. Some of the changes are

An improved UI. In addition to a cleaner, sleeker look, the UI now has a progress indicator and on-screen test descriptions. There is also a Simplified Chinese version of the UI.
Test automation. WebXPRT 2015 lets you automate testing, giving labs more flexibility and making it easier to test lots of devices.
New and improved tests. In addition to enhancing the existing tests, WebXPRT 2015 adds two new tests, Explore DNA Sequencing and Sales Graphs.

If you haven’t checked out the new WebXPRT, now is the time!

And remember the design document for the next generation of MobileXPRT should be out by the end of the month. If there are things you would like to see, it’s a great time to let us know.

Eric

Posted in Android, Automation, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, MobileXPRT, WebXPRT |

Category: Future of performance evaluation

Another great year

Nebula Wolf

Impressed and Excited

Upping our game

Something old, something new

WebXPRT 2015

Check out the other XPRTs: