Category: Application-based benchmarks

One benchmark to test them all

on April 7, 2016

It’s no secret that the XPRTs are a great way to get device results you can count on. Tens of thousands of people over six continents have used the XPRTs to help them make smart buying choices, and over a thousand media outlets have quoted XPRT results when reporting on the hottest tech. WebXPRT has always been the “go to” XPRT, because you can use it to test the widest range of devices. WebXPRT runs in the browser, however, so browser performance influences the results.

For a long time, our members and others have asked for a tool that would let you compare application performance on any type of device. People want a cross-platform XPRT that runs on devices the same way apps do.

We’re excited to announce that we’re going to create just that tool! Specifically, we’re going to create a version of MobileXPRT that runs on Android, iOS, and Windows.

This will not be easy. At one point in my career, I was in charge of a group that ported applications between platforms, and I learned from hands-on experience that doing that job well is very difficult. It’s not enough to simply make the application run; it also has to run efficiently on each type of system. MobileXPRT works at the application level, so we’ll have to deal with the many differences in the operating system architectures and APIs. We’ll have to make sure the code runs well on all three target OSes.

We’re willing to do all this work because the need for such a tool has never been greater. More and more devices hit the market all the time, and choosing the ones you want is tougher than ever. iPhone or Android phone? Windows tablet, Android tablet, or iPad?

The coming MobileXPRT will let buyers around the world answer those questions.

We’re not going to do this work in isolation. We will reach out to the OS vendors, because we want their input, comments, and help. We’ll make the source available to them, and we welcome their critiques and guidance in creating the best possible version for each OS.

Of course, we very much want your input, too. Do you have any thoughts about what you’d like to see in a cross-platform XPRT? If so, let us know!

Bill

Last week in the XPRTs
We published the XPRT Weekly Tech Spotlight on the Apple iPhone SE.
We added one new MobileXPRT ’15 result.
We added seven new WebXPRT ’15 results.

Posted in Android, Application-based benchmarks, Benchmark metrics, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, iOS, MobileXPRT, Performance benchmarking, WebXPRT, Windows 10 |

It’s show time!

By Eric Hale

on May 1, 2014

Tomorrow is the big day for BatteryXPRT 2014 for Android. After months of development, it’s finally time to release it to the world. We’ve learned quite a bit throughout the process, and we expect BatteryXPRT to become a trusted tool for Android users around the world.

BatteryXPRT will be available for download at BatteryXPRT.com, or through the Google Play Store. It will be available to the public no later than 5:00 PM ET on Friday May 2.

The BatteryXPRT results page allows you to submit test results directly from the app. We hope you’ll use this feature to submit your test runs for publication in the database.

There’s a huge diversity of Android devices out there, so BatteryXPRT.com will link to a page of tips, tricks, and known issues that we compiled during development testing. As with any of the XPRTs, if you encounter any issues please let us know.

Along with the new app, we also invite you to check out the Exploring BatteryXPRT 2014 for Android white paper for a more in-depth discussion of how it all works.

It’s definitely BatteryXPRT’s time to shine, but don’t forget that the new and improved TouchXPRT is coming soon!

Eric

Comment on this post in the forums

Posted in Android, Application-based benchmarks, Battery life, BatteryXPRT 2014 for Android, TouchXPRT 2014, White papers |

There is such a thing as too much

By Eric Hale

on December 6, 2012

There’s been a lot of excitement about TouchXPRT recently. However, we haven’t been ignoring HDXPRT. On November 9, we released a patch that lets HDXPRT support Windows 8. We’ve now integrated the patch into HDXPRT2012, so all copies of HDXPRT 2012 going forward will install on Windows 8 without the need for a separate step.

As promised, we will be releasing the source code for HDXPRT 2012. We anticipate having it available for community members by December 14.

During the comment period for HDXPRT, this message came through loud and clear: HDXPRT 2012 is too big and takes too long to run. So we are working hard to find the best way to reduce the number of applications and scenarios. While we want to make the benchmark smaller and faster, we want to make sure that HDXPRT 2013 is comprehensive enough to provide useful performance metrics for the greatest number of people.

We’re working toward having an RFC in late January that will define a leaner, meaner HDXPRT 2013, and will reflect the other comments we have as received as well. If you have thoughts about which applications and scenarios are most important to you, please let us know.

In other news, CES is coming in January, and Principled Technologies will be there! Once again, Bill is hoping to meet with as many of you in the Development Community as possible. We’ll have a suite at the Hilton and would love for you to come, kick back, and talk about HDXPRT, TouchXPRT, the future of benchmarks, or about the cool things you’ve seen at the show. (Bill loves talking about gadgets. Last year, he went into gadget overload!)

If you plan to be at CES, but are stuck working a booth or suite, let us know and Bill will try to stop by and say hi. Drop us an email at hdxrpt_CES@principledtechnologies.com and we will set up an appointment.

Finally, we’re really excited about the big changes at the Principled Technologies Web site. The new Web site gives us a lot of opportunities. Over the next few weeks, we’ll be looking at ways the Development Community can take advantage of them.

Eric

on this post in the forums

Posted in Application-based benchmarks, Benchmarking, CES, Collaborative benchmark development, HDXPRT 2013, HDXPRT Development Community benefits, Principled Technologies, What makes a good benchmark?, Windows 8 | Tagged CES |

Looking for a winner

By Eric Hale

on October 11, 2012

This week, PT published its first two public reports using HDXPRT 2012: Performance comparison: Dell Latitude E5430 vs. HP ProBook 4440s and Performance comparison: Dell Latitude E5430 vs. Lenovo ThinkPad L430. You should check them out.

Of course, you can find the HDXPRT results from these reports in the HDXPRT 2012 results database along with results from the characterization study we did last month. The results database is a repository of HDXPRT results you can use to compare system performance. The database includes full disclosure information and lets you sort by a number of criteria, including any HDXPRT score, the processor, amount of RAM, graphic card, and so on.

Looking at the results in the database got me wondering who has the mightiest machine out there. The current winner is a custom-built system with an Intel Core i7 3770 and 8 GB of RAM. It has an HDXPRT 2012 Create HD score of 248.

Records are meant to be broken, and I know someone out there can grind that score to dust. So, we’re going to have a contest. The first person to submit a set of HDXPRT results with a score above 248 will win at least bragging rights and maybe a prize if we can find something suitable around our offices.

You’ll find instructions for submitting results at Submit your HDXPRT 2012 results.

I can’t wait to see your results!

Eric

Comment on this post in the forums

Posted in Application-based benchmarks, Benchmarking, HDXPRT 2012, HDXPRT 2012 results, Performance benchmarking |

History in the making

By Bill Catchings

on July 13, 2012

We are quickly approaching the debut of HDXPRT 2012. It will be the second version of HDXPRT developed under the benchmark development community paradigm. This milestone provides a nice opportunity to look back at what has happened over the nearly two years since we started creating community-based benchmarks.

The most obvious accomplishment is the development of HDXPRT 2011 and HDXPRT 2012. HDXPRT 2011 has been used around the world for evaluating the performance of computers using applications doing activities that consumers do to create and consume content. We are hopeful that HDXPRT 2012 will be even more widely used.

We also announced earlier this year a new benchmark, TouchXPRT. This benchmark will provide a way to evaluate the performance of the emerging touch-based devices, including tablets. TouchXPRT will debut later this year, initially on Windows 8 Metro.

We have been working hard to get the word out about the benchmarks. We’ve been writing this weekly blog, conducting Webinars, and generally talking with folks in the computer industry. We’ve visited with members of the community around the world at trade shows like CES in Las Vegas and Computex in Taipei. We also spent time with members of the press and computer hardware and software developers. Over the coming months, we are planning to revamp the Web site, add video content, and generally find ways to better engage with and extend the development community.

Less obvious, but equally important to me, has been the development of the development community itself. Developing benchmarks has not been done this way before. We are doing what we can to make the process open to the community, including releasing the benchmark source code. We are optimistic that this method will grow and be a real asset for the industry.

As we look at the growing family of benchmarks under the benchmark XPRT umbrella, the question is always what is next? How can we improve the products and the community? What performance areas do we need to look at in the future? Battery life? Macs? Phones?

Thanks so much for joining us on this journey. The members of this community are what make it work. We look forward to continuing the journey with you!

Bill

Comment on this post in the forums

Posted in Application-based benchmarks, Collaborative benchmark development, HDXPRT, HDXPRT 2012, History of benchmarking, TouchXPRT |

Counting down

By Eric Hale

on July 6, 2012

We’ve been hard at work since the end of the beta period, driving toward the release of HDXPRT 2012. Things are looking good. The RTM is coming soon, so we thought we’d share the next few milestones with you.

RTM candidate: 7/13/12. At this point, we stop development. There are no feature changes after this point.
HDXPRT 2012 launch: 7/27/12. Having tested the RTM and manufactured the DVDs, we mail the benchmark to the community members. This is when the press release goes out. It’s also when we publish the HDXPRT 2012 white paper. Unlike the design document, this paper will explain HDXPRT 2012 to the general public.
Webinar: 8/3/28 – We talk about HDXPRT 2012 and take your questions.
First scaling paper: 8/31/12. As with HDXPRT 2011, we will be publishing a set of scaling studies. The first will test several modern desktop processors, looking at factors such as varying the amount of RAM, comparing HDDs to SSDs, and comparing internal graphics to a couple of popular graphics cards.
Second scaling paper: 9/28/12. In this paper, we expand on the testing we did in the first scaling paper.

We will release an update of HDXPRT 2012 that supports Windows 8 and includes bug fixes within a month of Windows 8 launch.

We’re very excited about the release of HDXPRT 2012, and look forward to seeing what you do with it!

Eric

Comment on this post in the forums

Posted in Application-based benchmarks, Collaborative benchmark development, HDXPRT, HDXPRT 2012, HDXPRT 2012 beta, HDXPRT development process, HDXPRT release cycle |

Category: Application-based benchmarks

One benchmark to test them all

It’s show time!

There is such a thing as too much

Looking for a winner

History in the making

Counting down

Check out the other XPRTs: