BenchmarkXPRT Development Community Blog

Discussing the future

on October 12, 2011

As I mentioned a couple weeks back, we’ve begun the HDXPRT 2012 development cycle by asking for suggestions. Before we get much further in the process, we’d like an opportunity to talk with you more directly. The best way to do that is via a webinar.

I’ll be hosting an HDXPRT webinar this coming Friday, October 14 at 2:00pm EST. I’m hoping you’ll be able to join us. I’ll go over the progress we’ve made this year in the HDXPRT Development Community, talk about the HDXPRT 2011 source code we recently released to members, and discuss the roadmap for the development cycle of HDXPRT 2012 that is just starting.

We really want your feedback on the current benchmark and your input on future directions. I encourage you to attend the webinar and let us know your ideas and suggestions. I expect the webinar will last about 45 minutes, depending upon the questions people have. We will be sending out email invitations to members shortly. If you have not joined the community, please do so now (at http://www.hdxprt.com/forum/register.php) and we will get you an invitation as well.

We know you may be unable to make the scheduled time, so we’ll post the webinar when it is over, as we have done in the past (http://hdxprt.com/webinar).

We value your input and participation in the HDXPRT benchmark process and look forward to your joining us later this week for the webinar. “See” you there!

Bill

Comment on this post in the forums

Posted in HDXPRT Webinar |

Looking back at the Mac

By Bill Catchings

on October 6, 2011

I expect that pretty much everyone is writing a blog/tribute to Steve Jobs. I’ll leave that to people more eloquent than I and to those who knew him. His death, however, made me think back to 1984 and the first computer I ever purchased with my own money—a 128KB original Macintosh. I was a mainframe and minicomputer systems programmer by training but had moved to CP/M and other personal computers. I had used a Lisa at work and had spent a bit of time on a Xerox Star, so the Mac’s UI concepts were very familiar to me. There was something special, however, about having it at home to play with. It was amazing to be able to use a word processor (MacWrite) that actually showed the fonts on the screen rather than having to embed Scribe (or even worse, TeX) commands into a text file to specify font changes. It was almost magical drawing things in MacPaint. None of what I drew was art, but it was fun. Even my two-year-old daughter found it fun to draw on the Mac with a mouse. A two-year-old using a mouse is nothing special today, but it was then.

I programmed on that early Mac using a C cross-compiler. With other folks at Columbia University, I wrote a version of the Kermit file transfer protocol for the Mac. My officemate even wrote a great Scrabble game for it. The owner of Scrabble was not interested in licensing it for use on a computer. It was a different time.

Over the years, I upgraded my Mac to 512KB and added a 5MB hard drive. I played games like Dark Castle on it. But, as all computers do, it got old and slow. Eventually, I packed it up. I bought a succession of MS-DOS PCs, Windows PCs, Amigas, newer Macs, and all manner of devices. I turned my love of ever faster and better and smaller computing devices into a career. Evaluating and measuring their performance is something I still do today.

That first Mac, though, will always be special—will always bring back memories of when computing was magical. I still have it in my basement. The originally beige case is now a rather strange yellow. I think I need to go and see if it still turns on. I’d like to see that happy Mac face again…

Bill

Comment on this post in the forums

Posted in History of computers |

Source Code for HDXPRT 2011 is Now Available to Members

By Eric Hale

on October 1, 2011

The HDXPRT Development Community, which Principled Technologies (PT) administers, is pleased to announce the availability to its members of the source code for the HDXPRT 2011 performance test tool.

HDXPRT 2011 is a benchmark for evaluating the capabilities of PCs in performing a wide variety of common consumer digital media tasks. Members of the Development Community have significantly influenced this version of the benchmark by helping to prepare its design specification and by beta-testing release candidates.

The release of the source code for a widely available application-based benchmark such as HDXPRT marks a milestone in the benchmark ecosystem. By opening the benchmark’s source code to its members, the HDXPRT Development Community is hoping to increase the transparency of the benchmarking process and to stimulate the participation of industry players and the public in the definition, development, understanding, and auditing of the benchmark.

Community members can download the benchmark’s source code directly from the official Community Web site,http://www.hdxprt.com. (Members must first accept a license agreement that prevents them from releasing the code to the public.) Any non-members who would like the source code can join the Community for a very modest fee by going to http://www.hdxprt.com.

In addition to downloading the source code of the current version of the tool, HDXPRT Development Community members can influence the design of future versions of HDXPRT and can even participate in the coding and development of these versions. Membership in the Community also keeps you up to date on the latest information regarding the benchmark. To learn more, visit the official HDXPRT Development Community Web site,http://www.hdxprt.com, and look for the Community’s presence on Facebook and Twitter.

About HDXPRT
HDXPRT, the High Definition Experience & Performance Ratings Test, is a software tool for evaluating the capabilities of PCs at handling real-world digital media scenarios involving common consumer applications. It includes tests for popular consumer usage models such as high-definition video transcoding, high dynamic range (HDR) photo manipulation, Windows 7 Drag and Drop transcoding for portable media players, and HD Flash video playback.

About Principled Technologies, Inc.
Principled Technologies, Inc. is a leading provider of fact-based marketing and technology assessment services. The founders, Mark Van Name and Bill Catchings, have worked together in technology assessment for over 25 years. As journalists, they published over a thousand articles on a wide array of technology subjects. They created and led the Ziff-Davis Benchmark Operation, which developed such industry-standard benchmarks as Ziff Davis Media’s Winstone and WebBench. They have also co-founded or led several other technology testing firms, including ZD Labs, eTesting Labs, and VeriTest.

Principled Technologies, Inc. is located in Durham, North Carolina, USA. For more information, please visit http://www.principledtechnologies.com.

Company Contact
Eric Hale
Principled Technologies, Inc.
1007 Slater Road
Suite 300
Durham, NC 27703
ehale@principledtechnologies.com
www.principledtechnologies.com

Suggestions?

By Bill Catchings

on September 28, 2011

In the midst of releasing the source code and results for HDXPRT 2011, we are beginning to plan for HDXPRT 2012. To make HDXPRT 2012 as good as possible, we want your suggestions. For the next four weeks, we want to hear what you would like to see in HDXPRT 2012. Obviously, we probably won’t be able to get everything done, but now is the time to dream.

To get you thinking, here are some areas where you might like to see changes or improvements:

Applications. What applications would you like to see in HDXPRT 2012? Should we add or remove any? Do you know of applications that would help us look at performance in areas we haven’t touched?
Workload scenarios. What activities should the applications carry out? Would you like to see other use case scenarios? Why?
Metrics. Are the current metrics easy enough to understand? Can you suggest improvements?
Execution. Are there issues with how HDXPRT 2011 runs that you would like to see improved?
Installation. Would you like to see any changes in how HDXPRT installs?
UI. Is the UI clear enough? Should we provide more progress feedback?
Documentation. Does the documentation give you what you need to run and understand HDXPRT? Would you like to see more white papers and results analysis?

To help make the suggestion period as interactive as possible, please check out the forum, http://www.hdxprt.com/forum/forumdisplay.php?11-HDXPRT-2012-Suggestions, and post your suggestions there. You can also send your suggestions to hdxprtsupport@hdxprt.com.

Bill

Comment on this post in the forums

Posted in HDXPRT, Let us know your thoughts |

Sharing results

By Bill Catchings

on September 22, 2011

A few weeks back, I wrote about different types of results from benchmarks. HDXPRT 2011’s primary metric is an overall score. One of the challenges of a score, unlike a metric such as minutes of battery life, is that it is hard to interpret without context. Is 157 a good score? The use of a calibration, or base, system helps a bit, because if that system has a score of 100, then a 157 is definitely better. Still, two scores do not give you a lot of context.

To help make comparisons easier, we are releasing a set of results from our testing at http://hdxprt.com/hdxprt2011results. With the results viewer we’ve provided, you can sort the results on a variety of fields and filter them for matching text. We’ve include results from our beta testing and our results white papers.

We’ll continue to add results, but we want to invite members of the HDXPRT Development Community to do the same. We would especially like to get any results you have published on your Web sites. Please submit your results using this link: http://www.hdxprt.com/forum/2011resultsubmit. We’ll give them a sanity check and then include them in the results viewer. Thanks!

Bill

Comment on this post in the forums

Posted in HDXPRT, HDXPRT 2011 results, HDXPRT metrics |

Getting to the source

By Bill Catchings

on September 16, 2011

Many of the earliest benchmarks came in source code form. Dhrystone and many others relied on the compiler for optimization. In fact, some compilers even recognized the code and basically optimized it to a few lines of code that did nothing but return the result! Even some modern benchmarks, such as SPEC CPU and LINPACK, come in source code form.

The source code to application benchmarks, however, has not typically been available. Two of the leading benchmarks of the last twenty years, Winstone and SYSmark, were never available in source code form. The makers of those tools had good reasons for keeping the code private; we know, because led the creation of Winstone. Keeping code private protects your intellectual investment, can make it easier to hit development schedules, and provides many other advantages.

It also, however, can lead some people to criticize that the reason you’re not showing the source code is that it is in some way biased. In benchmarks as in so many areas, transparency is the best way to allay such concerns.

Which leads us to today’s big announcement

We want HDXPRT to be as open as possible, so we’re bucking the normal practice for application-based benchmarks and planning to make the HDXPRT 2011 source code available to the HDXPRT Development Community.

The code will include both the benchmark harness and the scripts that drive the applications. You’ll be able to study everything about the benchmark. You’ll also be able to more easily contribute new code. Which is exactly what we hope you’ll do. We want you not only to be completely comfortable with the benchmark, we want you to contribute to future versions of it.

There will, of course, be some ground rules. We are making the code available only to the HDXPRT Development Community. (If you’re not already a member, joining is cheap and easy: just go here.) Because we want to limit the code to the community, to get access to it, members will have to agree to a license agreement that prevents them from releasing it to the public.

We don’t have an exact schedule in place yet, but over the next week or two, we should have all the necessary things in place to make the source code available.

When you’ve had a chance to look at it, please let us know what improvements you would like to see in HDXPRT 2012. We’ll discuss that version, and how you can help, in the coming weeks.

Bill

Comment on this post in the forums

Posted in Benchmarks in general, HDXPRT, HDXPRT Development Community benefits, HDXPRT source code |

Discussing the future

Looking back at the Mac

Source Code for HDXPRT 2011 is Now Available to Members

The HDXPRT Development Community, which Principled Technologies (PT) administers, is pleased to announce the availability to its members of the source code for the HDXPRT 2011 performance test tool.

Suggestions?

Sharing results

Getting to the source

Check out the other XPRTs: