BenchmarkXPRT Development Community Blog

Loose ends

on March 21, 2013

As we mentioned last week, the Samsung debuted the Galaxy S4 this past week. It seems to have hit all the expectations – including eye-scrolling. Looking at the reviews, it’s somewhere between the greatest phone ever made and something not quite as good as the iPhone. I’m looking forward to seeing one for myself and hoping someone submits a WebXPRT score for it soon.

We’ll be releasing the HDXPRT 2013 design document tomorrow. As we’ve said, the number one comment has been that HDXPRT 2012 is too big and takes too long to run. We have put a lot of thought into how to trim HDXPRT 2013 and still keep the essential value of the benchmark. We’ve also received some other good feedback that we’re incorporating, such as making installing and running HDXPRT scriptable.

We’ve been doing some investigation during the RFC period, and we’ve encountered problems scripting some of the applications, notably iTunes and PowerDirector. We’re working to overcome these problems but if they prove to be insurmountable, we might have to change the list of applications.

Again, thanks to everyone who commented on the HDXPRT 2013 RFC.

We’ve learned that some WebXPRT users in mainland China are having problems with very slow downloads. The good news is that results from the runs are valid. However, we understand that this is frustrating and are investigating solutions. If you are in China and have experienced slow downloads, please send an e-mail to benchmarkxprtsupport@principledtechnologies.com. We would like your help in evaluating any solutions we come up with.

Speaking of WebXPRT, over the next few weeks we’ll be releasing several white papers that look at WebXPRT results in more depth. The first paper will explain the WebXPRT confidence interval and how it relates to run to run variability, which has been confusing to a number of people. The second paper will look at the effect on the browser on WebXPRT scores. The third paper will look at the influence of the operating system.

Eric

Comment on this post in the forums

Posted in HDXPRT 2012 RFC, HDXPRT 2013, HDXPRT 2013 design document, WebXPRT, WebXPRT 2013 results |

Soon, we’ll know the truth

By Eric Hale

on March 14, 2013

I’m writing this on the morning of one of the most anticipated events in tech, Samsung’s rollout of the new Galaxy S4 phone. There have been many leaks, and it will be interesting to see how many are true. One feature generating buzz would let you scroll just by moving your eyes. Samsung filed for the patent, but there are conflicting reports about whether eye scrolling made it into the Galaxy S4.

The leaked reports say that the U.S. version of the Galaxy S4 will use Qualcomm Inc.’s quad-core chip, while versions for other markets will use Samsung’s Exynos 5 Octa-core chip. As we continue development of PhoneXPRT, I will be very interested to see how the different processors stack up performance wise.

For browser-level performance, WebXPRT 2013 is available today. If you get a new Galaxy S4, try out WebXPRT 2013 on it. As it says on the WebXPRT.com page, your run is confidential. However, anyone who shares a Galaxy S4 score in the next week gets a free t-shirt (until they are gone). You can tag BenchmarkXPRT on a Facebook post, use the hashtag #WebXPRT on Twitter, or email us at BenchmarkXPRTsupport@principledtechnologies.com.

In other news, the HDXPRT 2013 comment period has ended and we’re working on the design document. We’ll be sharing that with the community next week. If you have comments you haven’t sent in, please do so, and we’ll try to address them in the design document.

Eric

Comment on this post in the forums

Posted in HDXPRT 2012 RFC, HDXPRT 2013, PhoneXPRT, WebXPRT |

No time to rest

By Eric Hale

on March 7, 2013

Last week was possibly our biggest week ever. We announced PhoneXPRT, a new benchmark for evaluating the performance of smartphones, and released TouchXPRT 2013 and WebXPRT 2013 to the general public.

Since then, there’s been a lot of interest. The numbers keep going up! We’re not just talking about page views – people are downloading TouchXPRT and lots of people are running WebXPRT.

People have also been downloading the TouchXPRT source, which is very exciting. We strongly encourage community members to look at how the benchmark is put together. If you have programming skills and want to submit code, get in touch with us at benchmarkxprtsupport@principledtechologies.com. We’d love to hear from you!

Oh, and there’s a new video! This one introduces WebXPRT to the public. It gives a good idea of the range of devices WebXPRT will run on.

That’s a lot for one week. We’re not resting on our laurels, though. Obviously, we’re working on PhoneXPRT. However, let’s not forget about HDXPRT. The comment period for HDXPRT 2013 officially closed on March 6 and we are starting to work on the HDXPRT 2013 design document. If you have any feedback you haven’t sent, please do send it on. We’ll do our best to incorporate it into the design document.

Eric

Comment on this post in the forums

Posted in HDXPRT 2013, HDXPRT 2013 RFC, PhoneXPRT, TouchXPRT 2013 release, TouchXPRT 2013 source, Video, WebXPRT, WebXPRT 2013 release |

What a week!

By Eric Hale

on February 28, 2013

This has been quite a week for the BenchmarkXPRT family. We kicked off the week by announcing a new benchmark: PhoneXPRT. PhoneXPRT is designed to test small form factor devices, such as smart phones. PhoneXPRT will initially run on Android. For more information, read the press release at http://www.prweb.com/releases/phoneXPRT/pt0213/prweb10461639.htm

There was more to come. Today we formally released TouchXPRT 2013 and WebXPRT 2013. The community and the media have been using these as community previews for weeks now. Now that we’ve released the benchmarks, anyone may freely use them. You can read the press release at http://www.prweb.com/releases/TouchXPRTWebXPRT/0213/prweb10474803.htm

We have also released the TouchXPRT 2013 source to the community. The instructions for building the benchmark are the same as for building the community preview. Remember that community members have access to the source, but it is not available to the general public.

As WebXPRT 2013 moves from being a community preview to a public release, people may have concerns about privacy. While anyone using WebXPRT 2013 agrees to share their results, the database does not store any identifying information.

Remember, the HDXPRT comment period is still open! Please send your comments in. If you’ve not read the RFC yet, you can find it at http://www.principledtechnologies.com/hdxprt/forum/hdxprt2013RFC.php.

Eric

Comment on this post in the forums

Posted in PhoneXPRT, TouchXPRT 2013 release, TouchXPRT 2013 source, WebXPRT 2013 release, WebXPRT 2013 results |

TouchXPRT and WebXPRT benchmarks now available!

By Eric Hale

on February 28, 2013

The BenchmarkXPRT Development Community, which Principled Technologies (PT) administers, is pleased to announce the release of the TouchXPRT 2013 and WebXPRT 2013 benchmarks.

TouchXPRT 2013 is a benchmark for evaluating the capabilities of Windows 8 and Windows RT devices. TouchXPRT runs tests based on five user scenarios (Beautify photo album, Prepare photos for sharing, Convert videos for sharing, Export podcast to MP3, and Create slideshow from photos) and produces results for each of the five test scenarios plus an overall score.

WebXPRT 2013 is a browser-based multi-platform benchmark that uses representative Web-user scenarios to compare the performance of almost any Web-enabled device. It contains four HTML5- and JavaScript-based workloads: Photo Effects, Face Detect, Stocks Dashboard, and Offline Notes. Because the workloads are all HMTL5 based, they run on a wide variety of devices and operating systems—from iPad tablets to Android phones to Windows computers. In addition to scores, WebXPRT 2013 gives a report on the HTML5 capabilities of a device.

The benchmarks are now available to all from http://www.TouchXPRT.com andhttp://www.WebXPRT.com. Prior to today’s public release, the benchmarks were available exclusively to members of the BenchmarkXPRT Development Community and were used in multiple media reviews to evaluate a wide range of devices.

Members of the BenchmarkXPRT Development Community significantly influenced the development of the benchmarks by providing feedback on the initial design specifications and participating in beta testing. To learn how you can participate in the development of future versions of our benchmarks and stay up to date on the latest information, visit our Web site or look for us on Facebook and Twitter.

About the BenchmarkXPRT Development Community
The BenchmarkXPRT Development Community is a forum through which interested parties can contribute to the ongoing evolution of the BenchmarkXPRT suite of benchmarks. Community members play a vital role in ensuring that benchmarks such as HDXPRT, TouchXPRT, and WebXPRT incorporate the latest in the field of user-experience benchmarking and meet the needs and interests of a range of individuals and organizations.

About Principled Technologies, Inc.
Principled Technologies, Inc. is a leading provider of fact-based marketing and technology assessment services. The founders, Mark Van Name and Bill Catchings, have worked together in technology assessment for over 25 years. As journalists, they published over a thousand articles on a wide array of technology subjects. They created and led the Ziff-Davis Benchmark Operation, which developed such industry-standard benchmarks as Ziff Davis Media’s Winstone and WebBench. They have also co-founded or led several other technology testing firms, including ZD Labs, eTesting Labs, and VeriTest.

Principled Technologies, Inc. is located in Durham, North Carolina, USA. For more information, please visit http://www.principledtechnologies.com.

Company Contact
Eric Hale
Principled Technologies, Inc.
1007 Slater Road
Suite 300
Durham, NC 27703
ehale@principledtechnologies.com
www.principledtechnologies.com

Posted in BenchmarkXPRT Press Releases, Press Releases, TouchXPRT Press Releases, WebXPRT Press Releases |

New benchmark to evaluate smartphone performance

By Eric Hale

on February 25, 2013

Principled Technologies (PT) and the BenchmarkXPRT Development Community, which PT administers, are pleased to announce a new a benchmark, PhoneXPRT 2013.

This benchmark, which the community will be developing, will evaluate the capabilities of small form-factor devices. “We’re excited to be announcing this exciting new initiative during Mobile World Congress 2013,” said Bill Catchings, Chief Technology Officer of PT. “As popular as smartphones already are, the market is still developing, and today’s offerings differ widely in performance and responsiveness. The new benchmark will provide a fair and consistent basis for comparing the growing range of smartphones on the market.”

Community feedback will shape the new benchmark. Like all the BenchmarkXPRT benchmarks, PhoneXPRT 2013 will be based on real-world user scenarios and provide user-relevant metrics and results. The first operating system it will support is Android, but the intent is to support additional operating systems in the future.

“We invite all interested parties to participate in the creation, evaluation, and testing of this new benchmark,” Catchings said. Anyone can participate in the Development program by registering as a member of the BenchmarkXPRT Development Community. Community members have the opportunity to help shape future versions of the benchmarks. To register for the BenchmarkXPRT Development Community and begin participating in the creation process of the new benchmark, go to http://www.benchmarkxprt.com and follow the instructions in the ‘Join us’ section of the right-hand navigation menu.

To see the latest information on the benchmark’s development, visit http://www.phonexprt.com.

About Principled Technologies, Inc.
Principled Technologies, Inc. is a leading provider of fact-based marketing and technology assessment services. The founders, Mark Van Name and Bill Catchings, have worked together in technology assessment for over 25 years. As journalists, they published over a thousand articles on a wide array of technology subjects. They created and led the Ziff-Davis Benchmark Operation, which developed such industry-standard benchmarks as Ziff Davis Media’s Winstone and WebBench. They have also co-founded or led several other technology firms including ZD Labs, eTesting Labs, and VeriTest.

Principled Technologies, Inc. is located in Durham, North Carolina, USA. For more information, please visit http://www.principledtechnologies.com.

Company Contact
Eric Hale
Principled Technologies, Inc.
1007 Slater Road
Suite 300
Durham, NC 27703
ehale@principledtechnologies.com
www.principledtechnologies.com

Posted in BenchmarkXPRT Press Releases, MobileXPRT Press Releases, Press Releases |

Loose ends

Soon, we’ll know the truth

No time to rest

What a week!

TouchXPRT and WebXPRT benchmarks now available!

New benchmark to evaluate smartphone performance

Check out the other XPRTs: