PT-Logo
Forgot your password?
BenchmarkXPRT Blog banner

Category: Benchmarking computing devices

Question we get a lot

“How come your benchmark ranks devices differently than [insert other benchmark here]?” It’s a fair question, and the reason is that each benchmark has its own emphasis and tests different things. When you think about it, it would be unusual if all benchmarks did agree.

To illustrate the phenomenon, consider this excerpt from a recent browser shootout in VentureBeat:

 
While this looks very confusing, the simple explanation is that the different benchmarks are testing different things. To begin with, SunSpider, Octane, JetStream, PeaceKeeper, and Kraken all measure JavaScript performance. Oort Online measures WebGL performance. WebXPRT measures both JavaScript and HTML 5 performance. HTML5Test measures HTML5 compliance.

Even with benchmarks that test the same aspect of browser performance, the tests differ. Kraken and SunSpider both test the speed of JavaScript math, string, and graphics operations in isolation, but run different sets of tests to do so. PeaceKeeper profiles the JavaScript from sites such as YouTube and FaceBook.

WebXPRT, like the other XPRTs, uses scenarios that model the types of work people do with their devices.

It’s no surprise that the order changes depending on which aspect of the Web experience you emphasize, in much the same way that the most fuel-efficient cars might not be the ones with the best acceleration.

This is a bigger topic than we can deal with in a single blog post, and we’ll examine it more in the future.

Eric

What’s in a name?

A couple of weeks ago, the Notebookcheck German site published a review of the Huawei P8lite. We were pleased to see they used WebXPRT 2015, and the P8 Lite got an overall score of 47. This week, AnandTech published their review of the Huawei P8lite. In their review, the P8lite got an overall score of 59!

Those scores are very different, but it was not difficult to figure out why. The P8lite comes in two versions, depending on your market. The version Notebookcheck used is based on HiSilicon’s Kirin 620, while the version AnandTech used was Qualcomm’s Snapdragon 615 SoC. It’s also worth noting that the phone Notebookcheck tested was running Android 5.0, while the phone AnandTech tested was running Android 4.4. With different hardware and different operating systems, it’s no surprise that the results were different.

One consequence of the XPRTs being used across the world is that is that it is not uncommon to see results from devices in different markets. As we’ve said before, many things can influence benchmark results, so don’t assume that two devices with the same name are identical.

Kudos to both AnandTech and Notebookcheck for their care in presenting the system information for the devices in their reviews. The AnandTech review even included a brief description of the two models of the P8lite. This type of information is essential for helping people make informed decisions.

In other news, Windows 10 launched yesterday. We’re looking forward to seeing the TouchXPRT and WebXPRT results!

Eric

Device or computer?

As you may have noticed, I am fascinated by performance.  I’m also an avid cyclist and techno geek.  The recent start of the Tour de France has turned my thoughts to the technology of bikes and their accessories.  As with most technology, the latest models promise to be faster, lighter, and better.

One accessory of particular interest to me is the bike “computer.”  When I first started serious riding six years ago, bike computers were pretty minimal devices.  They were generally a small LCD display that connected via wires to two sensors.  One sensor counted how quickly a magnet on the pedal passed by to determine the cyclist’s cadence (pedal strokes per minute).  The other sensor counted how quickly a magnet on one of the wheels passed by. Knowing the circumference of the wheel, it calculated the cyclist’s speed and distance traveled.  Sure there had to be a processor of some sort in those bike computers, but I always refused to call it a bike computer. Speedometer/odometer seemed more accurate to me.

Now, however, I have on my bike a Garmin Edge 500 (https://buy.garmin.com/shop/shop.do?cID=160&pID=36728).  It is a small device—less than 2 inches by 3 inches—that attaches to my handle bars and determines my speed and distance via a built-in GPS. It determines altitude by detecting changes in barometric pressure and temperature by a built-in thermometer.  It communicates wirelessly with my heart rate monitor.  It can also talk wirelessly to other devices, like a cadence sensor or a power meter that measures the power applied to the pedals.  The LCD screen is customizable and allows me to display the information I most care about while riding.  The Edge 500 collects all of the data and can upload it via a computer to the Garmin Connect Web site.

By any definition of computer, the Edge 500 seems to qualify.  I still don’t call it a computer, however. Calling it a speedometer/odometer would be silly.  I tend to refer to it as my Garmin.  The line between computer and device is definitely getting blurrier.

We are all surrounded by more and more computing devices, whether they are desktops, notebooks, tablets, smart phones, or bike computers.  On some of those, performance is critical while on others, fast enough is all we care about.  On which devices do you think performance is important?  Even as we start the work on HDXPRT 2012, we are constantly examining other areas and types of devices that need benchmarks.  Let us know your thoughts!

Bill

Comment on this post in the forums

Check out the other XPRTs: