Category: HDXPRT development process

More HDXPRT 2012 changes

on May 11, 2012

We’re in the home stretch of testing the HDXPRT 2012 before releasing the beta to the community members. As is often the case, we’ve run into some issues that I want to share with you.

We’ve encountered some technical challenges in creating the 4K video playback workload that the design specification called for. We could not use the existing HDXPRT playback automation and instrumentation because Windows Media Player currently does not by default play 4K H.264 content. We tried other video players but ran into stability issues on slower systems.

These technical problems led us to step back and think about the role of video playback in HDXPRT 2012. In HDXPRT 2011, there wasn’t much differentiation between most current systems on the video playback results. The video playback tests were also a source of confusion. As I traveled and talked to users, they thought that the stars of the video playback were the primary metric, not the performance number. More generally, however, having both quality and performance numbers can be confusing.

Our plan is to remove the video playback tests from the beta. We will include them as optional inspection tests in an updated version later this year. By removing video playback from the primary tests and metrics, we’ll be able to include stressful test that won’t work on all systems while avoiding unnecessary confusion. Over time, I think we can include a number of interesting tests this way. We’d love to hear your ideas for what might be good to include that way as well as whether you think we’re making the correct call for video playback tests in HDXPRT 2012.

I will be traveling next week. Eric Hale, our project lead for HDXPRT 2012, will be writing the next blog. He will hopefully have some good news about the beta!

Bill

Comment on this post in the forums

HDXPRT 2012 characterization study

By Bill Catchings

on May 4, 2012

For HDXPRT 2011, we did quite a bit of testing to characterize the benchmark. Those results appeared in an initial white paper and a follow up one. In the first, we ran tests on different processors, various amounts of RAM, internal vs. external graphics, hard disk vs. SSD, and the effects of Intel Turbo Boost Technology. In the follow up white paper, we looked in more depth at the effect of graphics cards with different processors.

I mentioned a couple weeks ago that we are starting to put together a testbed to help us characterize HDXPRT 2012. What is in that testbed will define what characteristics of the upcoming benchmark we measure. We would like to get your help defining that testbed.

Our current thinking is to do a similar set of tests this year with updated hardware. However, plenty of additional things would be interesting to look at. First, I would like to increase the range of processors we test, including AMD processors. I would also like to do some testing varying different processor characteristics such as threads, cores, and frequency. It might also be good to look at the effect of new technologies like hybrid drives (which combine a small SSD with a hard disk to try and have the best of both).

We face two challenges in doing these characterization tests. One is to try and change one only variable at a time. That is very difficult in some cases, such as comparing Intel and AMD processors—you can’t just swap them in the same motherboard. Fortunately, it is usually possible to find very similar motherboards and keep other components (like disks, graphics, and RAM) constant. The other challenge is getting all of the necessary hardware in house.

So, we have two requests for you. First, let us know what you would like to see us test. Second, help us by supplying some of that equipment. If you supply the equipment we will do our best to include results from it in the characterization study and in the new HDXPRT 2012 results database. As always, thanks for your help!

Bill

Comment on this post in the forums

Posted in HDXPRT 2012, HDXPRT development process, HDXPRT workloads, Let us know your thoughts |

Change is inevitable

By Bill Catchings

on April 27, 2012

As we get close to the beta version of HDXPRT 2012, I wanted to let you know how it compares with the original design specification. As inevitably happens in any software project, there are differences between the original design and the final product. Generally, things have stayed pretty close with HDXPRT 2012, but there are two changes worth noting.

First, in the design specification, we specified Audacity 1.3.14 Beta in the Music Maker scenario as that was the only version that supported Windows 7 at the time. Audacity 2.0 with Windows 7 debuted in the interim so we are using that version.

The second and more significant issue was with Picasa, which was to be part of the Media Organizer scenario. Unfortunately, we couldn’t create a stable script because scripting tools like AutoIT could not properly recognize some of Picasa’s application UI elements. Somewhat reluctantly, we ended up replacing Picasa with Photoshop Elements. We still think the scenario is a good one and Photoshop Elements is an appropriate tool. I would have liked, however, to have Picasa in there.

There are probably some more minor differences between the beta and the design specification. We’ll let you know what they are when we have the beta ready in a couple weeks. (Hopefully!) We’re looking forward to getting that into your hands and getting your feedback. If you’re not already a member of the Development Community, I encourage you to join so that you can get your copy of the beta when it is available.

Bill

Comment on this post in the forums

Posted in HDXPRT 2012, HDXPRT 2012 design specification, HDXPRT development process |

HDXPRT 2012 – Testing the test

By Bill Catchings

on April 20, 2012

We’re currently starting testing on alpha versions of HDXPRT 2012. In order to do that, we’re putting together a testbed. We have two goals for the testbed that are somewhat contradictory. The first is to make the testbed as diverse as possible in terms of vendors and configurations. We want notebooks and desktops from as many vendors as possible. We want to make sure we have systems that will push the edges—both slower systems that may even be below the minimum recommended configuration and faster ones representing the current latest and greatest. These systems will help us shake out bugs and provide some raw data that we can publish when the benchmark debuts in the new results database.

The second goal for the testbed is to have systems where we can easily change one variable at a time to help us understand the characteristics of the benchmark. Typically, these are white box systems where we can swap processors, disks, RAM, and so on. We will use the results from these systems in the benchmark characterization white paper we will create for the debut of HDXPRT 2012.

We’d like your opinions on what we should be certain to test. We think we have a good handle on what to include, but we want your ideas as well.

We also are looking for additional systems to include in our testbed. If you can supply some, please let me know. That is one way to make sure HDXPRT 2012 works on your system and to get your results in the results database. Rest assured, we will not publish those results without your permission. Regardless, the more systems we can test, the better the final product will be.

There will, of course, be opportunities for you to help with the testing as we get to the beta stage in the near future.

Bill

Comment on this post in the forums

Posted in HDXPRT 2012, HDXPRT development process, Let us know your thoughts |

Quick HDXPRT 2012 status update

By Bill Catchings

on February 17, 2012

Between talking about CES, the new touch benchmark, and sausage making, it seems like it’ has been a while since I’ve said anything about HDXPRT. Here’ is a quick status update. The short form is that folks are heads- down coding, debugging, and testing. We still have some significant hurdles to overcome, such as trying to script Picassa. We also are going to have to make some difficult decisions in the near future about possibly swapping out one or two of the applications due to either licensing or scripting issues. (Sausage making at its best!) We’ll keep you posted in the forums when we have to make those decisions.

There is still a lot to get done, but things still appear to be on schedule. That schedule means that we are still hoping to have a beta version available for the Development Community to test in late March. At that point, the beta version will be available to our members and we will really need your help to try and shake things out. (Join at http://hdxprt.com/forum/register.php if you are not yet a member of the Development Community and want to help in our effort.) The more different systems and configurations we can all test together, the better the benchmark will be. There will also be at least some time for feedback on whether HDXPRT 2012 matches the design specification and if there are any last- minute tweaks you think would help make for a better benchmark.

So, stay tuned! We look forward to continuing to work with you on making HDXPRT 2012 even better than the current version.

Bill

Comment on this post in the forums

Posted in HDXPRT 2012, HDXPRT Development Community benefits, HDXPRT development process |

An open, top-down process

By Bill Catchings

on October 20, 2011

We’ve been hard at work putting together the RFC for HDXPRT 2012. As a group of us sat around a table discussing what we’d like to see in the benchmark, it became clear to me how different this development process is from those of other benchmarks I’ve had a hand in creating (3D WinBench, Winstone, WebBench, NetBench, and many others.). The big difference is not in the design or the coding or even the final product.

The difference is the process.

A sentiment that came up frequently in our meeting was “Sure, but we need to see what the community thinks.” That indicates a very different process than I am used to. Different from what companies developing benchmarks do and different from what benchmark committees do. What it represents, in a word, is openness. We want to include the Development Community in every step of the process, and we want to figure out how to make the process even more open over time. For example, we discussed ideas as radical as videoing our brainstorming sessions.

Another part of the process I think is important is that we are trying to do things top-down. Rather than deciding which applications should be in the benchmark, we want to start by asking how people really use high-definition media. What do people typically do with video? What do they do to create it and how do they watch it? Similarly, what do people do with images and audio?

At least as importantly, we don’t want to include only our opinions and research on these questions; we want to pick your brains and get your input. From there, we will work on the workflows, the applications, and the RFC. Ultimately, that will lead to the scripts themselves. With your input and help, of course!

Please let us know any ideas you have for how to make the process even more open. And tell us what you think about this top-down approach. We’re excited and hope you are, too!

Bill

Comment on this post in the forums

Posted in Benchmarks in general, HDXPRT 2012, HDXPRT development process, Let us know your thoughts |

Category: HDXPRT development process

More HDXPRT 2012 changes

HDXPRT 2012 characterization study

Change is inevitable

HDXPRT 2012 – Testing the test

Quick HDXPRT 2012 status update

An open, top-down process

Check out the other XPRTs: