BenchmarkXPRT Development Community Blog

Art or sausage?

on February 10, 2012

I discussed in my previous blog how weighing the tradeoffs between real science and real world in benchmark is a real art. One person felt it was more akin to sausage making than art! In truth, I have made that comparison myself.

That, of course, got me thinking. Is the process of creating a benchmark like that of creating sausage? With sausage, the feeling is that if you knew what went into sausage, you probably wouldn’t eat it. That may well be true, but I would still like to know that someone was inspecting the sausage factory. Sausage that contains strange animal parts is one thing, but sausage containing E. coli is another.

We are trying with the Development Community to use transparency to create better benchmarks. My feeling is that the more inspectors (members) there are, the better the benchmark will be. At least to me, unlike making sausage, creating benchmarks is actually cool. (There are probably sausage artisans who feel the same way about sausage.)

What do you think? Would you prefer to know what goes into making a benchmark? We hope so and hope that is why you are a part of this community. If you are not part of the Development Community, we encourage you to join at http://hdxprt.com/forum/register.php. Come join us in the sausage-making art house!

Bill

Comment on this post in the forums

The real art of benchmarking

By Bill Catchings

on February 3, 2012

In my last blog entry, I noted the challenge of balancing real-world and real-science considerations when benchmarking Web page loads. That issue, however, is inherent in all benchmarking. Real world argues for benchmarks that emphasize what users and computers actually do. For servers, that might mean something like executing real database transactions against a real database from real client computers. For tablets, that might mean real fingers selecting and displaying real photos. There are obvious issues with both—setting up such a real database environment is difficult and who wants to be the owner of the real fingers driving the tablet? It is also difficult to understand what causes performance differences—is it the network, the processors, or the disks in the server? There are also more subtle challenges, such as how to make the tests work on servers or tablets other than the original ones. Worse, such real-world environments are subject to all sorts of repeatability and reproducibility issues.

Real science, on the other hand, argues for benchmarks that emphasize repeatable and reproducible results. Further, real science wants benchmarks that isolate the causes of performance differences. For servers, that might mean a suite of tests targeting processor speed, network bandwidth, and disk transfer rate. For tablets, that might mean tests targeting processor speed, touch responsiveness, and graphics-rendering rate. The problem is that it is not always obvious what combination of such factors actually delivers better database server performance or tablet experience. Worse, it is possible that testing different databases and transactions would result in very different characteristics that these tests don’t at all measure.

The good news is that real world and real science are not always in opposition. The bad news is that a third factor exacerbates the situation—benchmarks take real time (and of course real money) to develop. That means benchmark developers need to make compromises if they want to bring tests to market before the real world they are attempting to measure has changed. And, they need to avoid some of the most difficult technical hurdles. Like most things, that means trying to find the right balance between real world and real science.

Unfortunately, there is no formula for determining that balance. Instead, it really is somewhat of an art. I’d love to hear from you some examples of benchmarks (current or from the past) that you think do a good job implementing this balance and showing the real art of benchmarking.

Bill

Comment on this post in the forums

Posted in Benchmarking, Benchmarks in general, What makes a good benchmark? |

Web benchmarking challenges

By Bill Catchings

on January 27, 2012

I think that an important part of any touch benchmark will be a Web component. After all, the always (or almost always) connected nature ofthese devices is a critical part of their identities. I think such a Web benchmark needs to include a measurement of page load speed (how long it takes to download and render a page).

Creating such a test seems straightforward. Pick a set of sites, such as the five or ten most popular, and then time how long the home page of each takes to load. The problem, however, is that those pages are constantly changing. Every few months, most popular sites do a major redesign. That would obviously affect the results for a test and make it difficult to compare the results of a current test to one from a few months back. It is even more of a problem that the page will be different for one user than another as sites typically know things like where you are and what your computer is and adjust things to match those characteristics. And, the ads and the content of the site are constantly changing and updating. Even hitting Refresh on a page can give you different page.

Given all of those problems, how is it possible to test page loads? One way is to create pages that are similar those of leading Web sites in terms of things like size, amount of graphics, and dynamic elements. This allows the tests to be consistent over time and from different devices and locations. (Or, at least, as consistent as the variability of the Internet from moment to moment allows.) The problem with this approach, however, is that the pages will age out as Web sites update themselves and they will not be the real sites.

Such are the tradeoffs in benchmarking. The key is how to balance real science with real world considerations. What do you think? Which approach is the better balance of real science and real world?

Bill

Comment on this post in the forums

Posted in Touch-based benchmarking |

Who is on board?

By Bill Catchings

on January 20, 2012

While talking with people at CES about HDXPRT and the upcoming touch benchmark, I encountered the same question a few times—Who are the current members of the Development Community? My answer was something along the lines of “About 10 PC hardware vendors, about the same number of press people, and a few other folks from companies around the world.” I was, however, itching to name the companies because the list is really pretty impressive. We haven’t asked for permission from the Development Community members, though, so I left my answer vague.

Given our goal of expanding the Development Community, I find myself weighing two possible outcomes if we were to make public the names of the companies represented. On the one hand, it could encourage others to join us (“All the other cool kids are doing it, I guess I will too!”). On the other hand, it could discourage others from joining us (“Not sure how my company would feel about this. Should I ask Legal? I’m too busy, never mind.”)

My best plan for now is to email each member individually and ask where he or she stands on company anonymity. And to give all new members the option of keeping their affiliation off the record. Rest assured that we will definitely not reveal this information without your permission.

We’d like to know what you think. Would you have joined the Development Community if doing so required identifying your company and allowing us to share it? Would you now be willing to let us say that someone from your company is a member?

Bill

Comment on this post in the forums

Posted in HDXPRT Development Community benefits, Let us know your thoughts |

CES: Gadget overload

By Bill Catchings

on January 13, 2012

I never thought I would say this, but there are more electronic gadgets and toys than I want. While walking the many cavernous show floors of CES, I saw cool bicycle gadgets from iBike (www.ibikedash.com). One device is a case for your iPhone that transforms it into a cycling computer. Because it measures wind speed, it actually is more capable than any existing bike computer—it uses data you supply like your type of bike and your weight, GPS info and knowledge of the terrain, and readings on wind speed and your heart rate to calculate your power output. If it works reliably, it would provide data that normally requires a cycle power meter costing a couple thousand dollars. If you are not into cycling, you probably don’t care, but it does show how our phones are becoming the gathering point for a myriad of data sources around us. I definitely need to try one of these out when they become available in March.

I also saw solar panels from Sharp (SunSnap) that have the inverter built in so that they output AC power directly. This gets around the messy inverter and wiring problems of typical panels that output DC power. Now, if I can get my homeowners association to agree, I need some of these.

I also saw TVs that were enormous, like 84-inch LCD, and gorgeous, like the 55-inch OLED, both from LG. I saw Windows 8 tablets and cars and iPhone cases and e-cigarettes. Basically, I reached gadget overload. At least the future of technology does not appear to be boring!

Thanks so much to the folks that stopped by our suite to talk about HDXPRT, the upcoming touch benchmark, and what they see as the future of benchmarking. We will be doing our best over the coming months to incorporate your ideas and suggestions. If you were not able to visit with us, please feel free to drop me an email and let me know what you are thinking.

Bill

Comment on this post in the forums

Posted in HDXPRT at CES, Let us know your thoughts |

A new benchmark

By Bill Catchings

on January 6, 2012

Hopefully, you caught our press release yesterday: http://www.hdxprt.com/touch_release.php. The short form of it is that we (PT and the HDXPRT Development Community) are going to create another benchmark—one aimed at touch-based devices. The obvious place to start will be Windows 8 Metro. Because it is a Windows-based touch platform, the development process should not be too different from work we have done in the past and may even allow us to use some code from HDXPRT. The longer-term goal of going cross-platform will be much more challenging. One step at a time!

Touch-based computing devices and Windows 8 Metro are emerging areas without any good tools for comparing performance. We are all excited at this opportunity to create a benchmark to meet that need and hope you are as well. After all, we will need your help to pull this off!

As I mentioned last week, I will be at CES. I would love to talk with as many of you as possible about the new touch benchmark and HDXPRT 2012. We’ll have a suite at the Hilton and I will be on the show floor checking out the latest gadgets, especially touch devices. Drop us an email at hdxprt_ces@hdxprt.com if you would like to stop by the suite or will be working a booth or suite and would like me to stop by. Either way, I’m hoping to see quite a few of you folks there.

Bill

Comment on this post in the forums

Posted in Touch-based computing, TouchXPRT |

Art or sausage?

The real art of benchmarking

Web benchmarking challenges

Who is on board?

CES: Gadget overload

A new benchmark

Check out the other XPRTs: