BenchmarkXPRT Blog banner

Category: Benchmarking

Chaos and opportunity

Eric Hale
By Eric Hale
on June 13, 2013

With both E3 and Apple’s WWDC happening this week, there’s been a lot of news. There’s also been a lot of hyperbolic commentary. I am not about to get into the arguments about the PS4 vs. the Xbox One or iOS 7 vs. Android.

It was Tim Cook’s presentation at WWDC that really got my attention. It’s unusual in an executive presentation to focus so much attention on a particular competitor, but Android was clearly on his mind. At one point, he focused harsh attention on fragmentation in the Android market, calling it “terrible” for developers. You can see the video here, at about 74 minutes.

As we saw in the 90s, chaos can breed innovation. At that time, the paradigm was that Macs always worked, but if you wanted the most advanced hardware, you should get a PC. I remember the editors at MacWorld, who deeply, truly loved the Mac, lusting over the (by the standards of the time) small, light, cheap notebooks PC users could get.

That being said, we understand the challenges of developing in the Android market. As I said in It’s finally here!, the Android ecosystem is sufficiently diverse that we know the benchmark will encounter configurations we’ve not seen before. If you have any problems with the MobileXPRT CP, please let us know at benchmarkxprtsupport@principledtechnologies.com. We want the benchmark to be the best it can be.

Eric

Comment on this post in the forums

Posted in Android, BenchmarkXPRT development community, History of benchmarking, History of computers, MobileXPRT |

The MobileXPRT 2013 community preview is available!

Eric Hale
By Eric Hale
on May 29, 2013

If you’re a member, you can download it here. If you’re not a member, join and see what the excitement’s about.

Comment on this post in the forums

Posted in Community Preview, MobileXPRT |

Presentation XPRT

Bill Catchings
By Bill Catchings
on May 9, 2013

A few weeks ago, the Mobile PC Extended Battery Life Working Group (EBL WG) asked me to give a presentation about the BenchmarkXPRT benchmarks during one of their meetings. The group started a little over 10 years ago and is an industry organization focusing on achieving all-day battery life in mobile PC platforms.

There was a catch, of course. The meeting was this week, in Taipei. Luckily, we were able to grant the request without racking up any frequent flier miles—thanks to the wonders of technology, I was able to give the presentation remotely.

Taipei is 12 hours ahead of us here on the East Coast, so I made arrangements to give the presentation last night at 9:30 pm, making it 9:30 am the next day there—the first thing on the agenda for today. (Oh, the vagaries of time zones!)

During the one-hour session we talked about HDXPRT, TouchXPRT, WebXPRT and MobileXPRT (which, for the moment, is how we internally refer to the XPRT formerly known as PhoneXPRT). And of course, we discussed the BenchmarkXPRT Development Community. It went well, and we had some great questions from the group.

It went so well, in fact, that I’d love to do it again. I like talking about the development community and the XPRTs, and would welcome the chance to present to other groups. If you are interested or know an organization that is, you can email us at benchmarkxprtsupport@principledtechnologies.com.

Bill

Comment on this post in the forums

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, HDXPRT, MobileXPRT, PhoneXPRT, TouchXPRT |

A happy coincidence

Bill Catchings
By Bill Catchings
on May 2, 2013

I love new gadgets and even the promise of new ones. Samsung just announced the specs for their upcoming Galaxy Tab 3. Initial reactions to the specs have been somewhat muted to say the least. Basically, they’ve looked at this as only an incremental improvement to the current model. The early rumors of a larger screen and bigger improvements, which turned out to be false, surely contributed to the disappointed reactions.

That being said, some sites claim that the performance of the Galaxy Tab 3 is much higher than the Galaxy Tab 2, particularly regarding graphics. We look forward to verifying these claims ourselves.

Coincidentally, this week we have been playing with an early version of PhoneXPRT (or whatever we end up calling it). So far, things are looking good. We ran it on several devices, including a Samsung Galaxy Tab 2. Like all the XPRT benchmarks, it uses real- world scenarios, which we think result in more useful and accurate results. We’ll talk more about the scenarios in the next few weeks.

It’s a very exciting time in the Android phone and tablet market. I can’t wait to try out subsequent versions of the new benchmark on the latest and greatest Android devices!

Bill

Comment on this post in the forums

Posted in MobileXPRT, Performance testing on tablets, PhoneXPRT, Touch-based devices |

Lies, damned lies, and statistics

Eric Hale
By Eric Hale
on April 11, 2013

No one knows who first said “lies, damned lies, and statistics,” but it’s easy to understand why they said it. It’s no surprise that the bestselling statistics book in history is titled How to Lie with Statistics. While the title is facetious, it is certainly true that statistics can be confusing—consider the word “average,” which can refer to the mean, median, or mode. “Mean average,” in turn, can refer to the arithmetic mean, the geometric mean, or the harmonic mean. It’s enough to make a non-statistician’s head spin.

In fact, a number of people have been confused by the confidence interval WebXPRT reports. We believe that the best way to stand behind your results is to be completely open about how you crunch the numbers. To this end, we released the white paper WebXPRT 2013 results calculation and confidence interval this past Monday.

This white paper, which does not require a background in mathematics, explains what the WebXPRT confidence interval is and how it differs from the benchmark variability we sometimes talk about. The paper also gives an overview of the statistical and mathematical techniques WebXPRT uses to translate the raw timing numbers into results.

Because sometimes the devil is in the details, we wanted to augment our overview by showing exactly how WebXPRT calculates results. The white paper is accompanied by a spreadsheet that reproduces the calculations WebXPRT uses. If you are mathematically inclined and would like to suggest improvements to the process, by all means let us know!

Eric

Comment on this post in the forums

Posted in Benchmark metrics, WebXPRT, WebXPRT 2013 results, White papers |

The HDXPRT 2013 RFC is here

Eric Hale
By Eric Hale
on February 14, 2013

We released the RFC, or request for comments, for HDXPRT 2013 yesterday. Our major objective with the RFC is to get your feedback. Your feedback played an important part in developing HDXPRT 2012, and we are hoping it plays an even larger role in developing HDXPRT 2013.

The RFC includes our thoughts and ideas for the design of HDXPRT 2013 based on the many conversations we’ve had over the six months since the current version of HDXPRT debuted. Indeed, during the last few weeks, we shared some of the feedback we received during and after the Webinar in January.

At this point, nothing is written in stone. Now is the time to let us know where you agree and where you disagree. For example, the current proposal drops support for Windows 7. Do you have an opinion about this? Let us know.

The RFC is available for Development Community members at http://www.principledtechnologies.com/hdxprt/forum/hdxprt2013RFC.php. Our goal is to get your feedback by March 6. We’d like as much of the feedback as possible to appear on the forums to help stimulate discussion. However, if you prefer to send in your comments via email, please send them to BenchmarkXPRTsupport@hdxprt.com.

Of course, you can send comments to us any time, and you don’t have to limit yourself to HDXPRT! Do you have thoughts about TouchXPRT or WebXPRT? They are both moving rapidly toward their official releases. Do you have thoughts about other benchmarks we should consider developing? Send those, too!

Eric

Comment on this post in the forums

Posted in Collaborative benchmark development, HDXPRT 2013, HDXPRT 2013 RFC, HDXPRT Development Community benefits, HDXPRT development process |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2025   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x