Month: July 2011

Always wanting to know more

on July 27, 2011

I’m an engineer (computer science) by training, and as a consequence I’m always after more data. More data means better understanding, which leads to better decision making. We acquired a lot of data in the course of finishing our white paper on the characteristics of HDXPRT 2011. Now, of course, I want even more.

The biggest area that I want to understand better is the graphics subsystem. Our testing showed processor-integrated graphics out-performing discrete graphics cards. That was not what I expected. There seem to be two likely explanations. The first is that since the workload of HDXPRT 2011 does not include 3D, discrete graphics cards are not that helpful to the benchmark’s applications. Certainly, 3D performance plays more to the traditional strengths of discrete graphics cards. The second likely explanation is that the integrated graphics on the second-generation Intel Core processors we used perform well. A number of performance Web sites have noted the same thing since the debut of those processors.

The answer is probably a combination of the two.

To satisfy my data desires, we’re going to look further. We’ll start by testing on some older processors as well as some different graphics cards. We’ll share our findings with you.

Please let us know any other characteristics of HDXPRT 2011 that you’d like us to explore in more depth. I can’t guarantee we’ll be able to look at everything, but I know I always want to know more!

Bill

Comment on this post in the forums

Sneak peak at the HDXPRT 2011 results white paper

By Bill Catchings

on July 20, 2011

After spending weeks testing different configurations with HDXPRT 2011, we are putting the final touches on a white paper detailing the results. I thought I’d give you a sneak peak at some of the things the tests revealed about the characteristics of HDXPRT 2011.

As I explained last week, trying to understand the characteristics of a benchmark requires careful testing while changing one component at a time. To do that, we ran the tests on a single system using an Intel DH67BL motherboard. We changed processors (both type and speed), the amount of RAM, the type of storage (hard disk and SSD), and the graphics subsystem, as well as a few other variables.

Here are a few of the things we found:

Processor speed – On an Intel Core i3, increasing the processor speed (GHz) 6.5% resulted in a 4.4% increase in the HDXPRT overall score. On an Intel Core i5, increasing the processor speed (GHz) 17.9% resulted in an 8.1% increase in the HDXPRT overall score. Generally, that means that increased processor speed is important, but the performance scales somewhat less than the raw gigahertz.

Memory – Increasing from 2 GB to 4 GB increased the overall score 10.7% on an Intel Core i5 and 15.8% on an Intel Core i7. However, increasing from 4 GB to 8 GB increased the score less than 2% on both processors. These results map pretty well with my personal experience: going to 4 GB is important for media-rich applications, but going to 8 GB is less so.

Disk drive – Switching from a hard disk to an SSD increased the overall score about 1%. While I would certainly prefer an SSD to a hard disk, this shows that, for HDXPRT 2011, disk performance has only a small influence on the results.

Many more details will be in the white paper we will publish in the next few days. Please be on the lookout for it and let us know what you think of the results and what they say about the characteristics of HDXPRT 2011.

We plan to conduct a Webinar in the near future to discuss the HDXPRT 2011 results white paper and to answer general questions. I hope to see you there!

Bill

Comment on this post in the forums

Posted in HDXPRT 2011 results, HDXPRT 2011 results whitepaper, HDXPRT Webinar, Let us know your thoughts |

Benchmarking a benchmark

By Bill Catchings

on July 13, 2011

One of the challenges of any benchmark is understanding its characteristics. The goal of a benchmark is to measure performance under a defined set of circumstances. For system-level, application-oriented benchmarks, it isn’t always obvious how individual components in the system influence the overall score. For instance, how does doubling the amount of memory affect the benchmark score? The best way to understand the characteristics of a benchmark is to run a series of carefully controlled experiments that change one variable at a time. To test the benchmark’s behavior with increased memory, you would take a system and run the benchmark with different amounts of RAM. Changing the processor, graphics subsystem, or hard disk lets you see the influence of those components. Some components, like memory, can change in both their amount and speed.

The full matrix of system components to test can quickly grow very large. While the goal is to change only one component at a time, this is not always possible. For example, you can’t change the processor from an Intel to an AMD without also changing the motherboard.

We are in the process of putting HDXPRT 2011 through a series of such tests. HDXPRT 2011 is a system-level, application-oriented benchmark for measuring the performance of PCs on consumer-oriented HD media scenarios. We want to understand, and share with you, how different components influence HDXPRT scores. We expect to release a report on our findings next week. It will include results detailing the effect of processor speed, amount of RAM, hard disk type, and graphics subsystem.

There is a tradeoff between the size of the matrix and how long it takes to produce the results. We’ve tried to choose the areas we felt were most important, but we’d like to hear what you consider important. So, what characteristics of HDXPRT 2011 would you like to see us test?

Bill

Comment on this post in the forums

Posted in Application-based benchmarks, Benchmark metrics, HDXPRT 2011 results, Let us know your thoughts, Performance benchmarking, What makes a good benchmark? |

Device or computer?

By Bill Catchings

on July 6, 2011

As you may have noticed, I am fascinated by performance. I’m also an avid cyclist and techno geek. The recent start of the Tour de France has turned my thoughts to the technology of bikes and their accessories. As with most technology, the latest models promise to be faster, lighter, and better.

One accessory of particular interest to me is the bike “computer.” When I first started serious riding six years ago, bike computers were pretty minimal devices. They were generally a small LCD display that connected via wires to two sensors. One sensor counted how quickly a magnet on the pedal passed by to determine the cyclist’s cadence (pedal strokes per minute). The other sensor counted how quickly a magnet on one of the wheels passed by. Knowing the circumference of the wheel, it calculated the cyclist’s speed and distance traveled. Sure there had to be a processor of some sort in those bike computers, but I always refused to call it a bike computer. Speedometer/odometer seemed more accurate to me.

Now, however, I have on my bike a Garmin Edge 500 (https://buy.garmin.com/shop/shop.do?cID=160&pID=36728). It is a small device—less than 2 inches by 3 inches—that attaches to my handle bars and determines my speed and distance via a built-in GPS. It determines altitude by detecting changes in barometric pressure and temperature by a built-in thermometer. It communicates wirelessly with my heart rate monitor. It can also talk wirelessly to other devices, like a cadence sensor or a power meter that measures the power applied to the pedals. The LCD screen is customizable and allows me to display the information I most care about while riding. The Edge 500 collects all of the data and can upload it via a computer to the Garmin Connect Web site.

By any definition of computer, the Edge 500 seems to qualify. I still don’t call it a computer, however. Calling it a speedometer/odometer would be silly. I tend to refer to it as my Garmin. The line between computer and device is definitely getting blurrier.

We are all surrounded by more and more computing devices, whether they are desktops, notebooks, tablets, smart phones, or bike computers. On some of those, performance is critical while on others, fast enough is all we care about. On which devices do you think performance is important? Even as we start the work on HDXPRT 2012, we are constantly examining other areas and types of devices that need benchmarks. Let us know your thoughts!

Bill

Comment on this post in the forums

Posted in Benchmarking computing devices, Let us know your thoughts |

Month: July 2011

Always wanting to know more

Sneak peak at the HDXPRT 2011 results white paper

Benchmarking a benchmark

Device or computer?

Check out the other XPRTs: