Category: Future of performance evaluation

In the spotlight

on January 14, 2016

I’m happy to be back in North Carolina, but I had a really great time at CES. I talked to over a dozen companies about the XPRTs and the XPRT Weekly Tech Spotlight, and had some good conversations. Hopefully, some of these companies’ devices will be among the first ones we showcase when the XPRT Weekly Tech Spotlight goes live next month.

Of course, I saw some really great tech at CES! Amazing TVs and cars, magic mirrors, all kinds of drones, and the list goes on. Before the show, the Internet of Things was predicted to be big this year, and boy, was it! Smart refrigerators, door locks, and thermostats were just the beginning. Some of my favorite examples were the chopsticks and footbath—both Bluetooth enabled—and “the world’s first remote controlled game shoe.”

Clearly IoT is the Wild West of technology right now. We’re had some conversations about how the XPRTs might be able to help consumers navigate the chaos. However, with a class of products this diverse, there are a lot of issues to consider. If you have any thoughts about this, let us know!

Eric

Posted in CES, Collaborative benchmark development, Future of performance evaluation, Internet of things, IoT, What makes a good benchmark?, XPRT Weekly Tech Spotlight |

Another great year

By Eric Hale

on December 31, 2015

A lot of great stuff happened this year! In addition to releasing new versions of the benchmarks, videos, infographics, and white papers, we released our first-ever German UI and sponsored our first student partnership at North Carolina State University. We visited three continents to promote the XPRTs and saw XPRT results published in six of them (we’re still working on Antarctica).

Perhaps most exciting, we reached our fifth anniversary. Users have downloaded or run the XPRTs over 100,000 times.

As great as the year has been, we are sprinting into 2016. Though I can’t talk about them yet, there are some big pieces of news coming soon. Even sooner, I will be at CES next week. If you would like to talk about the XPRTs or the future of benchmarking, let me know and we’ll find a time to meet.

Whatever your holiday traditions are, I hope you are having a great holiday season. Here’s wishing you all the best in 2016!

Eric

Posted in Benchmarking, Benchmarks in general, BenchmarkXPRT development community, CES, Collaborative benchmark development, Future of performance evaluation, German, History of benchmarking |

Nebula Wolf

By Eric Hale

on December 10, 2015

A couple of months ago, we talked about the senior project we sponsored with North Carolina State University. We asked a small team of students to take a crack at implementing a game that we could use as the basis of a benchmark test.

Last Friday, the project culminated with a presentation at the annual Posters and Pies event.

The team gave a great presentation, and I was impressed by how much they accomplished in 3 months. They implemented a game that they called Nebula Wolf – the mascot for NC State is a wolf. It’s a space-themed rail shooter. You can play the game, or click a button to run a script for benchmarking purposes. In the scripted mode, Nebula Wolf unlocks the frame rate so the device can run at full speed.

Over the next couple of weeks, we’re going to be testing Nebula Wolf, digging into the code and getting a deeper understanding of what the team did. We’re hoping to make the game available on our web site soon.

Tomorrow, AJ, Brien, and Rachel will present one last time, here at PT. It’s been a real pleasure working with them. I wish them all good luck as they finish college and start their careers.

Eric

Posted in Benchmark metrics, Benchmarking, Collaborative benchmark development, Future of performance evaluation |

Impressed and Excited

By Eric Hale

on September 17, 2015

A couple of weeks ago we talked about some initiatives we’ve been exploring. This week, we’re happy to be able to talk about a new project. We’re sponsoring a senior project with the computer science department at North Carolina State University (NCSU).

As part of their education, small teams of seniors work with local companies on various programming projects. The students are expected to put significant time into these projects, and the tasks aren’t easy. You can get a sense of the range and complexity of these projects by looking at past project proposals.

We submitted a proposal for creating an experimental benchmark test to NCSU and they accepted it and assigned us a team of students. I’ve met a couple of times now with them. I’m impressed and very excited about what they’re going to do. We’re not ready to talk about the project yet, but if they’re successful, we’ll make the new test available on the BenchmarkXPRT site. We might even include it in a future XPRT!

Our hope is that if this project is successful, we can replicate it at other schools and help train the next generation of benchmark developers. As a bonus, the BenchmarkXPRT community will get some fresh perspectives and some new experimental test tools.

Eric

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation |

Upping our game

By Eric Hale

on August 27, 2015

As we wrote last week, we’re releasing MobileXPRT 2015 to the public tomorrow. Thanks to everyone who helped make the community preview a success!

We’re working on the TouchXPRT 2016 design document and will make it available for the community to review soon.

As you know, we’re always investigating initiatives that could improve our game. We’re continuing to investigate creating experimental tests for future XPRTs. Experimental tests will allow us to maintain broad compatibility for each XPRT tool while providing testers with an opportunity to evaluate cutting-edge technologies.

Another initiative involves looking for new partnerships with people who are not yet part of the community, but could add valuable input to the development process. It’s too soon to say much more about this, but we’re having fruitful conversations and hoping that these partnerships will grow the community even more!

If you have ideas about experimental tests, improving the XPRTs, or expanding the community, please let us know.

Eric

Posted in Android, Collaborative benchmark development, Community Preview, Future of performance evaluation, MobileXPRT, MobileXPRT 2015, TouchXPRT, TouchXPRT 2016, What makes a good benchmark? |

Something old, something new

By Eric Hale

on June 25, 2015

Last week, we talked about porting TouchXPRT 2014 to be a Windows 10 universal app. This will let it run on devices running Windows 10 and those running Windows 10 mobile.

We won’t be retiring TouchXPRT 2014 when we release the Windows 10 universal app version. Windows 8 doesn’t support Windows 10 universal apps, but Windows 10 will be able to run Windows 8 applications. This means you’ll also be continue to be able to use TouchXPRT 2014 to test Windows 8 based systems, as well as to compare Windows 8 and Windows 10 performance.

The results from TouchXPRT 2014 and the universal app version of the benchmark won’t be compatible. Even though the test scenarios will be the same, the porting process means that we have to change the APIs the benchmark is using and rebuild the benchmark with different tools.

We’re currently debating changing the way we version the benchmarks. As the number of versions of each benchmark increases, it may make sense to move away from year-based versioning. This will obviously affect what we call the new Windows 10 version of TouchXPRT. If you have any thoughts on this, please let us know!

Eric

Posted in BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, TouchXPRT, TouchXPRT 2014, TouchXPRT release cycle, Windows 10, Windows 8 |

Category: Future of performance evaluation

In the spotlight

Another great year

Nebula Wolf

Impressed and Excited

Upping our game

Something old, something new

Check out the other XPRTs: