BenchmarkXPRT Development Community Blog

BenchmarkXPRT Development Community Releases Source Code for HDXPRT 2012

on February 7, 2013

The BenchmarkXPRT Development Community, which Principled Technologies (PT) administers, is pleased to announce the availability of the source code for the HDXPRT 2012 performance test tool.

HDXPRT 2012 is a benchmark for evaluating the capabilities of PCs in performing a wide variety of common consumer digital media tasks. Members of the development community have significantly influenced this version of the benchmark by helping to prepare its design specification and by beta-testing release candidates.

The release of the source code for a widely available application-based benchmark such as HDXPRT marks a milestone in the benchmark ecosystem. By opening the benchmark’s source code to its members, the BenchmarkXPRT Development Community is hoping to increase the transparency of the benchmarking process and to stimulate the participation of industry players and the public in the definition, development, understanding, and auditing of the benchmark.

Community members can download the benchmark’s source code directly from the official benchmark Web site, http://www.hdxprt.com. Any non-members who would like the source code can join the community for a very modest fee by going to http://www.benchmarkxprt.com.

In addition to downloading the source code of the current version of the tool, BenchmarkXPRT Development Community members can influence the design of future versions of HDXPRT and other benchmarks. They can even participate in the coding and development of these benchmarks. Membership in the community also keeps you up to date on the latest information. To learn more, visit the official BenchmarkXPRT Development Community Web site,http://www.benchmarkxprt.com, and look for the community’s presence on Facebook and Twitter.

About HDXPRT
HDXPRT, the High Definition Experience & Performance Ratings Test, is a software tool for evaluating the capabilities of PCs at handling real-world digital media scenarios involving common consumer applications. It
includes tests for popular consumer usage models such as high-definition video transcoding, and high dynamic range
(HDR) photo manipulation.

About the BenchmarkXPRT Development Community BenchmarkXPRT Development Community is a forum through which interested parties can contribute to the ongoing evolution of the BenchmarkXPRT suite of benchmarks. Community members play a vital role in ensuring that benchmarks such as HDXPRT, TouchXPRT, and WebXPRT incorporate the latest in the field of user-experience benchmarking and meet the needs and interests of a range of individuals and organizations.

About Principled Technologies, Inc.
Principled Technologies, Inc. is a leading provider of fact-based marketing and technology assessment services. The founders, Mark Van Name and Bill Catchings, have worked together in technology assessment for over 25 years. As journalists, they published over a thousand articles on a wide array of technology subjects. They created and led the Ziff-Davis Benchmark Operation, which developed such industry-standard benchmarks as Ziff Davis Media’s Winstone and WebBench. They have also co-founded or led several other technology testing firms, including ZD Labs, eTesting Labs, and VeriTest.

Principled Technologies, Inc. is located in Durham, North Carolina, USA. For more information, please visit http://www.principledtechnologies.com.

Company Contact
Eric Hale
Principled Technologies, Inc.
1007 Slater Road
Suite 300
Durham, NC 27703
ehale@principledtechnologies.com
www.principledtechnologies.com

Straight from the source

By Eric Hale

on February 7, 2013

One of the pillars of our community model of benchmark development is making the source available. As we’ve said many times, we believe that doing so leads to better benchmarks.

Today we released the source for HDXPRT 2012. As with previous versions of HDXPRT, the source is available only to community members, not to the general public. We apologize that it has taken so long. HDXPRT is complicated to build, and we wanted to have a simpler and more robust build process before we made the source available.

The source allows you to examine how HDXPRT is implemented and to try some experiments of your own. Because of the size of HDXPRT 2012, the source package does not include the applications or the data files for the workloads. By including only the benchmark source code and associated files, we could keep the package small enough to download. If you want to try some changes for experiments and test them, all you need to do is install HDXPRT 2012 from the distribution DVDs. The compilation instructions will tell you how to copy your modified executables over the shipping versions.

Community members can get instructions on how to download the source code here (registration required).

If you create something interesting while you’re experimenting, let us know! We’d love to have the community consider it for HDXPRT 2013.

Speaking of the community, we’ve sent T-shirts to all community members who’ve supplied their up-to-date mailing address. If you’re a community member who wants a shirt but hasn’t yet let us know, please e-mail benchmarkxprtsupport@principledtechnologies.com with your mailing address by February 15th.

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT, HDXPRT Development Community benefits, HDXPRT source code |

Keep them coming!

By Eric Hale

on January 31, 2013

Questions and comments have continued to come in since the Webinar last week. Here are a few of them:

How long are results valid? For a reviewer like us, we need to know that we can reuse results for a reasonable length of time. There is a tension between keeping results stable and keeping the benchmark current enough for the results to be relevant. Historically, HDXPRT allowed at least a year between releases. Based on the feedback we’ve received, a year seems like a reasonable length of time.
Is HDXPRT command line operable? (asked by a community member with a scripted suite of tests) HDXPRT 2012 is not, but we will consider adding a command line interface for HDXPRT 2013. While most casual users don’t need a command line interface, it could be very valuable to those of us using HDXPRT in labs.
I would be hesitant to overemphasize the running time of HDXPRT. The more applications it runs, the more it can differentiate things and the more interesting it is to those of us who run it at a professional level. If I could say “This gives a complete overview of the performance of this system,” that would actually save time. This comment was a surprise, given the amount of feedback we received saying that HDXPRT was too large. However, this gets to the heart of why we all need to be careful as we consider which applications to include in HDXPRT 2013.

If you had to miss the Webinar, it’s available at the BenchmarkXPRT 2013 Webinars page.

We’re planning to release the HDXPRT 2013 RFC next week. We’re looking forward to your comments.

Eric

Comment on this post in the forums

Posted in Benchmark metrics, Benchmarking, HDXPRT, HDXPRT workloads |

Some good questions

By Eric Hale

on January 24, 2013

On Tuesday, we had a Webinar for the BenchmarkXPRT community. This Webinar covered the material that Bill would have given in individual presentations at CES. As such, it was an overview of the XPRT family.

The Webinar was well attended. We will be posting the slides and the recording of the Webinar online soon. However, we got some good questions, and thought we’d share our responses with you.

How will updates to TouchXPRT, and other benchmarks, affect results? We will avoid affecting results as much as possible. However, when updates do affect results, we will disclose the effect and the testing we performed to verify it.

Will we provide a way for benchmark users to talk to each other about support issues, perhaps via OpenBlog? We had envisioned the benchmark forums providing this opportunity. However, we are very happy to look into ways to make community communication easier and more effective.

Do you provide company memberships, as opposed to individual memberships? Not currently, although we will certainly look into this. We have no formal voting mechanism, as SPEC and some other organizations have. We may get there one day, but it’s not currently an issue. If your concern is about paying multiple membership fees, contact us, and we’ll work with you to avoid that.

In HDXPRT, can you select the CPU or GPU for video conversion and control the quality of the conversion? We have not investigated this. HDXPRT installs the applications using the default settings. However, because HDXPRT installs the applications in a separate step from running the test, it might be possible to manually change the benchmark settings and then run HDXPRT. We will be looking into this and reporting on it going forward.

How does the server influence WebXPRT results? We have run WebXPRT hosted on different servers in different locations, and seen little influence on the results. However, as part of preparing the WebXPRT general release, we will characterize and document the influence of the server.

Feel free to let us know what you think about these or any other topics. As I said earlier, we’ll be posting the whole Webinar online soon.

Eric

Comment on this post in the forums

Posted in Benchmarking, Benchmarks in general, CES, HDXPRT, TouchXPRT, WebXPRT |

Unchecked ambition

By Bill Catchings

on January 17, 2013

With 10 applications and five scenarios, HDXPRT 2012 is the most ambitious version of HDXPRT ever. However, as we said in the blog post of the same name, There is such a thing as too much. We heard that message from you as well: HDXPRT 2012 is too big. At 11 GB, we can’t make it available for download, and it won’t fit on a single DVD.

It also takes longer to use than many of you would like. The installation takes a couple of hours, and each iteration takes 2 to 3 hours. Because a valid run of HDXPRT 2012 comprises three iterations, getting a single result takes all day.

As we work to reduce the size of HDXPRT 2013, we’re being careful. While we need to shrink the benchmark and its running time, we want to be sure that we don’t compromise its essential value. As part of this process, we’re looking at both the selection of applications and the ways we can reengineer the scenarios.

If you have ideas about HDXPRT 2012 that you haven’t sent yet, now is a good time to do that. We’ll be sending out the HDXPRT 2013 RFC in the next couple of weeks. Once it’s out, we will look forward to hearing your comments.

I also wanted to mention that as we had to cancel CES at the last minute, we’re planning to do a Webinar next week (Tuesday, January 22) to cover the material we planned to present there. We’ll send out an email later this week with more details.

Bill

Comment on this post in the forums

Posted in BenchmarkXPRT, CES, HDXPRT, HDXPRT 2013, HDXPRT Development Community benefits, Let us know your thoughts |

The umbrella is open

By Eric Hale

on January 11, 2013

Last week, we told you that we were going to be reorganizing the XPRT benchmarks under the BenchmarkXPRT umbrella. We’ve now taken a big step forward. The BenchmarkXPRT.com Web site is live.

BenchmarkXPRT.com is now the one place to go for news about all the benchmarks. When you visit the HDXPRT site, you’ll notice that the TouchXPRT and WebXPRT links are gone. The HDXPRT.com site is now specific to HDXPRT, and so matches the TouchXPRT.com and WebXPRT.com sites.

As part of this effort, we have also created a new Facebook page and Twitter feed for BenchmarkXPRT and they are already active. However, so that no one misses any important information, we will redundantly post the news to the old HDXPRT page and tweet to the old HDXPRT account until April 1.

The next step in the process will be converting the existing HDXPRT Development Community memberships to be BenchmarkXPRT Development community memberships. The new memberships will work like the ones you have now. However, you’ll also have the ability to follow specific benchmarks. That means that any time there’s news related to a specific benchmark, you’ll be notified. In the next couple of days, we will send an email to the members explaining this in more detail and giving the specific date of the cut over.

We will also be renaming the HDXPRT forums to be BenchmarkXPRT forums, and reorganizing the forums themselves.

As we’ve said before, this new organization better fits with the reality now, and gives us flexibility for the future. Who knows how when the family might grow again?

Eric

Comment on this post in the forums

Posted in BenchmarkXPRT, HDXPRT, Let us know your thoughts, Principled Technologies |

BenchmarkXPRT Development Community Releases Source Code for HDXPRT 2012

The BenchmarkXPRT Development Community, which Principled Technologies (PT) administers, is pleased to announce the availability of the source code for the HDXPRT 2012 performance test tool.

Straight from the source

Keep them coming!

Some good questions

Unchecked ambition

The umbrella is open

Check out the other XPRTs: