Category: Benchmarking

Putting the TouchXPRT pedal to the metal

on October 19, 2012

Since we announced TouchXPRT early this year, we’ve been following a typical benchmark development path. We started with the most important question—“What are people likely to do with a touch-based Windows 8 device?”—and built from there. We looked at what people are doing now with iOS- and Android-based devices. We worked with early Windows 8 units. We studied app stores. We spoke with members of the development community. And so on. When we were done studying, we moved to coding.

We’re making great progress, but something has been nagging at us: When Windows 8 tablets and other devices ship next week, there just won’t be much in the way of tools for measuring their performance when running Windows 8 apps. Sure, you may be able use standard benchmarks to assess the performance of typical desktop applications, but that won’t tell you how the devices will perform with tablet apps.

So, we’ve decided to put the pedal to the metal and provide everyone in our development community with a special treat. Sometime next week, before Windows 8 ships, we plan to release a sneak preview of TouchXPRT, the TouchXPRT 2013 Community Preview 1 (CP1).

CP1, as its name makes clear, is not the final TouchXPRT release. It is, though, a useful tool for beginning to measure Windows 8 device performance. It is also a great way for everyone in the community to see the current state of our thinking and to provide us feedback—rather than read a design spec, you can actually run this version of the tool and see what you think! (If you would like to read the informal design spec, check out http://www.hdxprt.com/forum/touchxprt2013cp1.php .)

To make the tool easier to evaluate and more useful to all of us, we’re also taking two more unusual steps:

1. We’re not putting any publication restrictions on this preview release. Test at will, and publish your findings.

2. We’re releasing the source code to all community members. If you’re curious about not just what we’re doing but how we’re doing it, you can find out.

We hope these steps will speed acceptance of TouchXPRT 2013 and foster more and faster feedback. Releasing a preview version is more work, because we have to do much of the work of a software release and on less-than-final code, but we believe the value to our community justifies the effort.

Next week, when we release CP1, I’ll go over more details, the known limitations, and how you can get us your feedback—feedback we very much want.

Between now and then, we’ll be readying CP1 for your use.

Bill

Comment on this post in the forums

Posted in Performance testing on tablets, Touch-based computing, TouchXPRT, TouchXPRT development process, TouchXPRT results |

Looking for a winner

By Eric Hale

on October 11, 2012

This week, PT published its first two public reports using HDXPRT 2012: Performance comparison: Dell Latitude E5430 vs. HP ProBook 4440s and Performance comparison: Dell Latitude E5430 vs. Lenovo ThinkPad L430. You should check them out.

Of course, you can find the HDXPRT results from these reports in the HDXPRT 2012 results database along with results from the characterization study we did last month. The results database is a repository of HDXPRT results you can use to compare system performance. The database includes full disclosure information and lets you sort by a number of criteria, including any HDXPRT score, the processor, amount of RAM, graphic card, and so on.

Looking at the results in the database got me wondering who has the mightiest machine out there. The current winner is a custom-built system with an Intel Core i7 3770 and 8 GB of RAM. It has an HDXPRT 2012 Create HD score of 248.

Records are meant to be broken, and I know someone out there can grind that score to dust. So, we’re going to have a contest. The first person to submit a set of HDXPRT results with a score above 248 will win at least bragging rights and maybe a prize if we can find something suitable around our offices.

You’ll find instructions for submitting results at Submit your HDXPRT 2012 results.

I can’t wait to see your results!

Eric

Comment on this post in the forums

Posted in Application-based benchmarks, Benchmarking, HDXPRT 2012, HDXPRT 2012 results, Performance benchmarking |

Make sure your voice is heard

By Eric Hale

on October 4, 2012

One thing about the community model we use for developing HDXPRT is that is depends on the community. Your input is essential to making the benchmark the best it can be. As the community grows, we’re learning more about your priorities.

During the development of HDXPRT 2012, we made the decision to remove the playback tests from the benchmark. While the design document called for the playback test to include 4K H.264, Windows Media Player does not play that format by default. Because less demanding codecs were not differentiating systems, and because the stars used to report the results confused some people, it seemed like a reasonable decision. Bill announced the decision in a blog post, More HDXPRT 2012 changes.

Fast forward to September 18, when Bill hosted the HDXPRT 2012 Webinar. During the Q&A session, a new member of the community said that the playback tests from HDXPRT 2011 were what got him interested in the benchmark. For now, he has to use HDXPRT 2011 for those tests, although, as per Bill’s original blog post, we may release a more demanding playback test as a standalone inspection test later this year.

The suggestion period for HDXPRT 2013 started on October 1. Now is the time to let us know what tests are the most useful to you. If there are tests you’d like us to add, tests you’d like us to change, applications you’d like us to consider, we need to know that too. You can post your suggestions to the forum in the HDXPRT 2013 Suggestions section or mail them to hdxprtsupport@hdxprt.com.

In November, we’ll develop an RFC for HDXPRT 2013 and send it to the community for review.

While the suggestions we receive early have the best chance of being implemented, comments we receive after the formal suggestion period still get our attention. We’re always listening. Contact us anytime and make sure that HDXPRT 2013 includes the things that are important to you.

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT 2012, HDXPRT 2013, HDXPRT Webinar, Let us know your thoughts |

Meeting the old fashioned way

By Mark Van Name

on August 30, 2012

As Bill discussed last week, we work in many different ways to build the community. Facebook, Webinars, Twitter, this blog—all of these social media venues help connect those of us in the community.

Sometimes, though, it’s nice to sit down in person and chat.

Which is precisely what I’m inviting fellow community members to do at the upcoming Intel Developer Forum (IDF). IDF runs from September 11 to 13 at the Moscone West Convention Center in San Francisco. While I’ll be busy attending the show for part of each day, I’d love to spend some of my free time talking with people about both HDXPRT and TouchXPRT. We could do it informally, by simply running across each other on the show floor, or we could schedule some meetings.

We could even make a group lunch of it. If a few folks can agree on a day and time, I’d be happy to set up and host a lunch at a nearby restaurant so we can share ideas while enjoying a meal.

If you’re interested, feel free to email me, and we’ll arrange to get together. I hope to have many good conversations with community members at the show!

Mark Van Name

Comment on this post in the forums

Posted in Benchmarking, Collaborative benchmark development, Let us know your thoughts |

Growing the community

By Bill Catchings

on August 23, 2012

Growing the community

It’s probably stating the obvious to say that what makes the XPRT Development Community work is the people. The more people there are in the community, the better the resulting benchmarks will be. Naturally, that means we want more people in the community.

Part of PT’s role in building the community is to get more people involved. Over the last couple of years, that has meant doing a number of different things—from conducting Webinars (we’ll have the next one in September) to meeting at trade shows (we expect to be at CES again next year) to contacting individuals in person and via email to using social media like Twitter (follow us at @HDXPRT) and Facebook (like HDXPRT Development Community). As I mentioned last week, we are now working on some videos to help get out the message. Basically, we’ve been telling everyone we can about the benchmarks and the community that creates them.

One thing we haven’t done, however, is to actively enlist the community itself in building the community. Here are some ways you can help. Please like, follow, or whatever us on the different social media platforms. Invite others to join the community. Probably most importantly, use the benchmarks and publish the results. Now that HDXPRT 2012 is out, we want to see its results popping up in as many places as possible.

You can also help by telling us your ideas for how to grow the community. What else do you think we should be doing?

Bill

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT 2012 |

Lights! Camera! Action!

By Bill Catchings

on August 10, 2012

While we continue working on the testing, white papers, and Webinars that are part of the HDXPRT 2012 roll-out, we also are working on some other things as well. The importance of HDXPRT shows how the world is moving toward a much more visual/graphical future. To keep up with that trend and with our benchmark, we are adding video content to the site.

At Principled Technologies, we have been embracing video in a big way. We built a dedicated video-production studio (picture lots of green walls) and bought the necessary video, audio, and lighting gear, so our talented video-production staffers have all the tools they need. Our goal is to be able to quickly create for our clients all sorts of videos of things like our performance reports. We are now able to utilize those same video assets for the XPRTs (HDXPRT and TouchXPRT).

To get things rolling here, we created a video blog based on an entry from a few weeks ago. We will be experimenting with doing more with video, including videos explaining and promoting HDXPRT and regular video blogs. Let us know what you would like to see!