Month: September 2011

Suggestions?

on September 28, 2011

In the midst of releasing the source code and results for HDXPRT 2011, we are beginning to plan for HDXPRT 2012. To make HDXPRT 2012 as good as possible, we want your suggestions. For the next four weeks, we want to hear what you would like to see in HDXPRT 2012. Obviously, we probably won’t be able to get everything done, but now is the time to dream.

To get you thinking, here are some areas where you might like to see changes or improvements:

Applications. What applications would you like to see in HDXPRT 2012? Should we add or remove any? Do you know of applications that would help us look at performance in areas we haven’t touched?
Workload scenarios. What activities should the applications carry out? Would you like to see other use case scenarios? Why?
Metrics. Are the current metrics easy enough to understand? Can you suggest improvements?
Execution. Are there issues with how HDXPRT 2011 runs that you would like to see improved?
Installation. Would you like to see any changes in how HDXPRT installs?
UI. Is the UI clear enough? Should we provide more progress feedback?
Documentation. Does the documentation give you what you need to run and understand HDXPRT? Would you like to see more white papers and results analysis?

To help make the suggestion period as interactive as possible, please check out the forum, http://www.hdxprt.com/forum/forumdisplay.php?11-HDXPRT-2012-Suggestions, and post your suggestions there. You can also send your suggestions to hdxprtsupport@hdxprt.com.

Bill

Comment on this post in the forums

Posted in HDXPRT, Let us know your thoughts |

Sharing results

By Bill Catchings

on September 22, 2011

A few weeks back, I wrote about different types of results from benchmarks. HDXPRT 2011’s primary metric is an overall score. One of the challenges of a score, unlike a metric such as minutes of battery life, is that it is hard to interpret without context. Is 157 a good score? The use of a calibration, or base, system helps a bit, because if that system has a score of 100, then a 157 is definitely better. Still, two scores do not give you a lot of context.

To help make comparisons easier, we are releasing a set of results from our testing at http://hdxprt.com/hdxprt2011results. With the results viewer we’ve provided, you can sort the results on a variety of fields and filter them for matching text. We’ve include results from our beta testing and our results white papers.

We’ll continue to add results, but we want to invite members of the HDXPRT Development Community to do the same. We would especially like to get any results you have published on your Web sites. Please submit your results using this link: http://www.hdxprt.com/forum/2011resultsubmit. We’ll give them a sanity check and then include them in the results viewer. Thanks!

Bill

Comment on this post in the forums

Posted in HDXPRT, HDXPRT 2011 results, HDXPRT metrics |

Getting to the source

By Bill Catchings

on September 16, 2011

Many of the earliest benchmarks came in source code form. Dhrystone and many others relied on the compiler for optimization. In fact, some compilers even recognized the code and basically optimized it to a few lines of code that did nothing but return the result! Even some modern benchmarks, such as SPEC CPU and LINPACK, come in source code form.

The source code to application benchmarks, however, has not typically been available. Two of the leading benchmarks of the last twenty years, Winstone and SYSmark, were never available in source code form. The makers of those tools had good reasons for keeping the code private; we know, because led the creation of Winstone. Keeping code private protects your intellectual investment, can make it easier to hit development schedules, and provides many other advantages.

It also, however, can lead some people to criticize that the reason you’re not showing the source code is that it is in some way biased. In benchmarks as in so many areas, transparency is the best way to allay such concerns.

Which leads us to today’s big announcement

We want HDXPRT to be as open as possible, so we’re bucking the normal practice for application-based benchmarks and planning to make the HDXPRT 2011 source code available to the HDXPRT Development Community.

The code will include both the benchmark harness and the scripts that drive the applications. You’ll be able to study everything about the benchmark. You’ll also be able to more easily contribute new code. Which is exactly what we hope you’ll do. We want you not only to be completely comfortable with the benchmark, we want you to contribute to future versions of it.

There will, of course, be some ground rules. We are making the code available only to the HDXPRT Development Community. (If you’re not already a member, joining is cheap and easy: just go here.) Because we want to limit the code to the community, to get access to it, members will have to agree to a license agreement that prevents them from releasing it to the public.

We don’t have an exact schedule in place yet, but over the next week or two, we should have all the necessary things in place to make the source code available.

When you’ve had a chance to look at it, please let us know what improvements you would like to see in HDXPRT 2012. We’ll discuss that version, and how you can help, in the coming weeks.

Bill

Comment on this post in the forums

Posted in Benchmarks in general, HDXPRT, HDXPRT Development Community benefits, HDXPRT source code |

Looking deeper into results

By Bill Catchings

on September 8, 2011

A few weeks ago, I mentioned some questions we had about graphics performance using HDXPRT 2011 after releasing our results white paper. The issue was that HDXPRT 2011 gave results I had not expected—the integrated graphics outperformed discrete graphics cards. I suspected that this was both because HDXPRT 2011’s lack of 3D work lessens the advantage of discrete graphics cards and because the integrated graphics on the second-generation Intel Core processors we used performed well.

We ran some tests with discrete graphics cards on an older processor (an Intel Core 2 Quad processor Q6600) and report our findings in a second results white paper. My suspicions were correct: On the older processor, the discrete graphics cards performed 21 to 36 percent better than the integrated graphics.

As an aside, we are looking into putting our test results on the Web site in some easy-to-access fashion so you can look at them in more detail. My hope is that doing so will facilitate sharing of results among all of us in the HDXPRT Development Community.

Based on this second results white paper, I would love to hear your responses to two questions. First, do you think that future versions of HDXPRT should include 3D graphics? Second, what other areas of HDXPRT 2011 would you like to see us look into?

Bill

Comment on this post in the forums

Posted in HDXPRT, HDXPRT 2011 results, HDXPRT 2011 results whitepaper, HDXPRT development process, Let us know your thoughts |

Month: September 2011

Suggestions?

Sharing results

Getting to the source

Looking deeper into results

Check out the other XPRTs: