Category: Collaborative benchmark development

Exploring virtual reality

on December 15, 2016

We’ve talked a lot in recent weeks about new technologies we are evaluating for the XPRTs. You may remember that back in June, we also wrote about sponsoring a second senior project with North Carolina State University. Last week, the project ended with the traditional Posters and Pies event. The team gave a very well thought‑out presentation.

As you can tell from the photo below, the team designed and implemented a nifty virtual reality app. It’s a room escape puzzle, and it looks great!

The app is a playable game with the ability to record the gameplay for doing repeatable tests. It also includes a recording that allows you to test a device without playing the game. Finally, the app lets you launch directly into the prerecorded game without using a viewer, which will be handy for testing multiple devices.

The team built the app using the Google Cardboard API and the Unity game engine, which allowed them to create Android and iOS versions. We’re looking forward to seeing what that may tell us!

After Posters and Pies, the team came to PT to present their work and answer questions. We were all very impressed with their knowledge and with how well thought out the application was.

Many thanks to team members Christian McCurdy, Gregory Manning, Grayson Jones, and Shon Ferguson (not shown).

Thanks also to Dr. Lina Battestilli, the team’s technical advisor, and Margaret Heil, Director of the Senior Design Center.

We are currently evaluating the app, and expect to make it available to the community in early 2017!

Eric

Posted in Android, Benchmarking, Collaborative benchmark development, iOS, North Carolina State University, Virtual reality |

BatteryXPRT’s future

By Justin Greene

on December 1, 2016

A few weeks ago, we discussed the future of HDXPRT. This week, we’re focusing on the current state of BatteryXPRT 2014 for Android, and how the benchmark may evolve in 2017.

BatteryXPRT continues to provide users with reliable evaluations of their Android device’s performance and battery life under real-world conditions. Originally designed to be compatible with Android 4.2 (Jelly Bean) and above, the benchmark continues to work well on subsequent versions of Android, up to and including Android 6.0 (Marshmallow).

Since Android 7 (Nougat) began to roll out on select devices in the past few months, our internal testing has shown that we’ll need to adjust the BatteryXPRT source code to maintain compatibility with devices running Android 7 and above. We developed the existing source when Eclipse was the officially supported SDK environment, and now we need to bring the code in line with the current Android Studio SDK.

Practically speaking, BatteryXPRT does run on Nougat, and to the best of our knowledge, battery life results are still accurate and reliable. However, the test will not produce a performance score. As more Nougat devices are released in the coming months, it’s possible that other aspects of the benchmark may encounter issues. If this happens during your testing, we encourage you to let us know.

Because MobileXPRT 2015 and BatteryXPRT 2014 performance workloads are so closely related, the next obvious question is whether MobileXPRT 2015 runs on Nougat devices. As of now, MobileXPRT 2015 does run successfully and reliably on Android 7, and this is because the most recent build of MobileXPRT 2015 was compiled using a newer SDK.

We think the best course of action for MobileXPRT 2015 and BatteryXPRT will be to eventually combine them into a single, easy-to-use Android benchmark for performance and battery life. We’ll talk more about that plan in the coming months, and we look forward to hearing your input. Until that transition is successful, we’ll continue to support both BatteryXPRT and MobileXPRT 2015.

As always, we welcome your feedback on these developments, as well as any ideas you may have for future XPRTs.

Justin

Posted in Android, Android L, Android M, Android N, Battery life, BatteryXPRT 2014 for Android, Collaborative benchmark development, MobileXPRT 2015 |

Machine learning

By Bill Catchings

on November 23, 2016

A couple months ago I wrote about doing an inventory of our XPRT tools. Part of that is taking a close look at the six existing XPRTs. The first result of that effort was what I recently wrote about HDXPRT. We’re also looking at emerging technology areas where the BenchmarkXPRT Community has expertise that can guide us.

One of the most exciting of these areas is machine learning. It has rapidly gone from interesting theoretical research (they called them “neural nets” back when I was getting my computer science degree) to something we all use whether we realize it or not. Machine learning (or deep learning) is in everything from intelligent home assistants to autonomous automobiles to industrial device monitoring to personalized shopping in retail environments.

The challenge with developing a benchmark for machine learning is that these are still the early days of the technology. In the past, XPRTs have targeted technologies later in the product cycle. We’re wondering how the XPRT model and the members of its community can play a role here.

One possible use of a machine-learning XPRT is with drones, a market that includes many vendors. Consumers, hobbyists, builders, and the companies creating off-the-shelf models could all benefit from tools and techniques that fairly compare drone performance.

The best approach we’ve come up with to define a machine-learning XPRT starts with identifying common areas such as computer vision, natural language processing, and data analytics, and then, within each of those areas, identifying common algorithms such as AlexNet, GoogLeNet, and VGG. We would also look at the commonly used frameworks such as Caffe, Theano, TensorFlow, and CNTK.

The result might differ from an existing XPRT where you simply run a tool and get a result. Instead, it might take the form of sample code and workloads. Or, maybe even one or two executables that could be used in the most common environments.

At this point, our biggest question is, What do you think? Is this an area you’re interested in? If so, what would you like to see a machine-learning XPRT do?

We’re actively engaging with people in these emerging markets to gauge their interest as well. Regardless of the feedback, we’re excited about the possibilities!

Bill

Posted in Collaborative benchmark development, Drones, Future of performance evaluation, HDXPRT, Machine learning |

HDXPRT’s future

By Bill Catchings

on November 3, 2016

While industry pundits have written many words about the death of the PC, Windows PCs are going through a renaissance. No longer do you just choose between a desktop or a laptop in beige or black. There has been an explosion of choices.

Whether you want a super-thin notebook, a tablet, or a two-in-one device, the market has something to offer. Desktop systems can be small devices on your desk, all-in-ones with the PC built into the monitor, or old-style boxes that sit on the floor. You can go with something inexpensive that will be sufficient for many tasks or invest in a super-powerful PC capable of driving today’s latest VR devices. Or you can get a new Microsoft Surface Studio, an example of the new types of devices entering the PC scene.

The current proliferation of PC choices means that tools that help buyers understand the performance differences between systems are more important than they have been in years. Because HDXPRT is one such tool, we expect demand for it to increase.

We have many tasks ahead of us as we prepare for this increased demand. The first is to release a version of HDXPRT 2014 that doesn’t require a patch. We are working on that and should have something ready later this month.

For the other tasks, we need your input. We believe we need to update HDXPRT to reflect the world of high-definition content. It’s tempting to simply change the name to UHDXPRT, but this was our first XPRT and I’m partial to the original name. How about you?

As far as tests, what should a 2017 version of HDXPRT include? We think 4K-related workloads are a must, but aren’t sure whether 4K playback tests are the way to go. What do you think? We need to update other content, such as photo and video resolutions, and replace outdated applications with current versions. Would a VR test would be worthwhile?

Please share your thoughts with us over the coming weeks as we put together a plan for the next version of HDXPRT!

Bill

Posted in Application-based benchmarks, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, HDXPRT, HDXPRT development process, Performance benchmarking, What makes a good benchmark?, Windows 10 |

Taking a detour

By Eric Hale

on October 13, 2016

Back in April, Bill announced that we would be starting development of a cross-platform benchmark. This announcement generated a lot of interest and we got lots of good feedback and ideas.

We knew from the start that getting a cross-platform benchmark right would be hard. However, it proved to be even trickier than we thought. As I explained before, benchmarks not only have to run well, but the results must be fair to all platforms involved. Achieving both of these requirements has been a challenge.

At the same time we’ve been devoting a great deal of effort and resources to the cross-platform benchmark, some increasingly popular new use cases have been receiving less attention than they deserve. We’ve decided that the cross-platform benchmark is not the best use of the Community’s resources, and are going to put it on the shelf for a while. This will free up the resources to let us really dig into some newer technologies.

Thanks again to everyone who responded.

Eric

Posted in Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Cross-platform benchmarks |

Rebalancing our portfolio

By Bill Catchings

on September 29, 2016

We’ve written recently about the many new ways people are using their devices, the growing breadth of types of devices, and how application environments also are changing. We’ve been thinking a lot about the ways benchmarks need to adapt and what new tests we should be developing.

As part of this process, we’re reviewing the XPRT portfolio. An example we wrote about recently was Google’s statement that they are bringing Android apps to Chrome OS and moving away from Chrome apps. Assuming the plan comes to fruition, it has big implications for CrXPRT, and possibly for WebXPRT as well. Another example is that once upon a time, HDXPRT included video playback tests. The increasing importance of 4K video might mean we should bring them back.

As always, we’re interested in your thoughts. Which tests do you see as the most useful going forward? Which ones do you think might be past their prime? What new areas do you like to see us start to address? Let us know!

Over the coming weeks, we’ll share our conclusions based on these market forces and your feedback. We’re excited about the possibilities and hope you are as well.

Bill

Posted in Android, Application-based benchmarks, BenchmarkXPRT development community, Chrome OS, Collaborative benchmark development, CrXPRT, Future of performance evaluation, HDXPRT, WebXPRT, What makes a good benchmark? |

Category: Collaborative benchmark development

Exploring virtual reality

BatteryXPRT’s future

Machine learning

HDXPRT’s future

Taking a detour

Rebalancing our portfolio

Check out the other XPRTs: