BenchmarkXPRT Development Community Blog

There is such a thing as too much

on December 6, 2012

There’s been a lot of excitement about TouchXPRT recently. However, we haven’t been ignoring HDXPRT. On November 9, we released a patch that lets HDXPRT support Windows 8. We’ve now integrated the patch into HDXPRT2012, so all copies of HDXPRT 2012 going forward will install on Windows 8 without the need for a separate step.

As promised, we will be releasing the source code for HDXPRT 2012. We anticipate having it available for community members by December 14.

During the comment period for HDXPRT, this message came through loud and clear: HDXPRT 2012 is too big and takes too long to run. So we are working hard to find the best way to reduce the number of applications and scenarios. While we want to make the benchmark smaller and faster, we want to make sure that HDXPRT 2013 is comprehensive enough to provide useful performance metrics for the greatest number of people.

We’re working toward having an RFC in late January that will define a leaner, meaner HDXPRT 2013, and will reflect the other comments we have as received as well. If you have thoughts about which applications and scenarios are most important to you, please let us know.

In other news, CES is coming in January, and Principled Technologies will be there! Once again, Bill is hoping to meet with as many of you in the Development Community as possible. We’ll have a suite at the Hilton and would love for you to come, kick back, and talk about HDXPRT, TouchXPRT, the future of benchmarks, or about the cool things you’ve seen at the show. (Bill loves talking about gadgets. Last year, he went into gadget overload!)

If you plan to be at CES, but are stuck working a booth or suite, let us know and Bill will try to stop by and say hi. Drop us an email at hdxrpt_CES@principledtechnologies.com and we will set up an appointment.

Finally, we’re really excited about the big changes at the Principled Technologies Web site. The new Web site gives us a lot of opportunities. Over the next few weeks, we’ll be looking at ways the Development Community can take advantage of them.

Eric

on this post in the forums

Posted in Application-based benchmarks, Benchmarking, CES, Collaborative benchmark development, HDXPRT 2013, HDXPRT Development Community benefits, Principled Technologies, What makes a good benchmark?, Windows 8 | Tagged CES |

TouchXPRT Web test update

By Bill Catchings

on November 29, 2012

On October 22, we released TouchXPRT CP1 to the community. We took the unprecedented step of releasing CP1 without any restrictions on publishing results, and since then reviews of the Microsoft Slate and the Sony Duo 11 Convertible Laptop have used TouchXPRT.

The five scenarios in CP1 focus on media manipulation. While this is an important activity on touch devices, we know this is not all people do.

Next week, we plan to release Web-based scenarios. They use HTML 5 for a variety of activities. Unlike the original scenarios in TouchXPRT CP1, there will be nothing to download. You simply browse to a URL and run the tests online. There’s nothing to set up, just browse and run.

That means that there is nothing preventing you from running these tests on pretty much any system-browser combination that supports HTML5, not just on touch-based, Windows 8 devices like the rest of TouchXPRT. That started us wondering whether these Web-based activities should be thought of as a different benchmark entirely. When these tests are available, please try them out and let us know what you think. Do you think they are worthwhile for a broader range of devices? Do you think their scenario-based emphasis is a good alternative to existing lower-level Web-based tests?

Please keep in mind that it’s not too late to give feedback on TouchXPRT CP1. Let us know how you like the scenarios on CP1 as well as what other activities you would like to see.

-Bill Catchings

Comment on this post in the forums

Posted in Benchmarking, HTML5, Performance testing on tablets, Touch-based benchmarking, Touch-based devices, TouchXPRT, Web-based testing |

Lots of things happening!

By Eric Hale

on November 15, 2012

If you’re interested in the interface formerly known as Metro (IFKaM), it’s an exciting time. A lot has happened in the last couple of weeks.

Perhaps the most exciting news is that the first Windows 8 tablets using the Intel Atom (Clover Trail) processor have started hitting the shelves, with many more coming in the next few weeks.

Nor are things are boring in the Windows RT space.

The first Windows RT tablet with LTE capabilities will be available this week from AT&T. You’ll find the press release here.
Microsoft released new firmware for Windows RT devices. Anecdotal evidence suggests that the firmware update improves performance. This is something we will be investigating in the near future.
Microsoft added Windows RT to its Compatibility Center. You can go there to find a comprehensive list of devices that are compatible with Windows RT. Because TouchXPRT’s scenarios focus on media creation and manipulation, we were particularly interested in cameras, media players, and TV devices.

With so much news, TouchXPRT needs some news of its own. When we released the community preview, we gave permission to publish results – the first time that’s happened for a benchmark that was not final. Now we are going to be creating an online database of results from the community preview. Look for it to go live next week.

With so many new devices and updated old ones, there’s a lot out there to test. We hope that you’ll share the results of your testing, and submit them to the database. The more results we get, the more valuable the database will be.

One final note: Microsoft released its first security patches for Windows 8 on Tuesday. If you are running Windows 8, whether on a touch device or not, check them out.

Eric

Comment on this post in the forums

Posted in Touch-based devices, TouchXPRT, TouchXPRT results, Windows 8 |

Back in September

By Eric Hale

on November 8, 2012

Back in September, we made a prerelease version of a patch to let HDXPRT run on Windows 8. (See Working towards Windows 8.) We’ve been testing the final version of the patch and hope to release it in the next few days.

We wanted to give you a heads up that there are a few differences between the versions of Windows that may affect your HDXPRT testing. For example, we have found that some that older systems that meet the minimum specs for HDXPRT 2012 and Windows 8 may not have Windows 8 drivers available. The lack of these drivers can cause the benchmark to fail.

Also, while it was possible for a lab like ours to test without activation of Windows 7 during the 30-day grace period, activation is different under Windows 8. To reliably run HDXPRT on Windows 8, activation is required.

We are updating the HDXPRT documentation based on our testing. Generally, though, testing on Windows 8 is very similar to testing on Windows 7.

We are releasing the patch on Friday, November 9 . Please try it out and let us know what you think!

Eric

Comment on this post in the forums

Posted in HDXPRT 2012, HDXPRT development process, Let us know your thoughts, Windows 8 |

Touch device first impressions

By Bill Catchings

on November 1, 2012

One of the nice benefits of working on TouchXPRT has been the opportunity to play with the latest touch devices. The latest two are Windows 8/RT tablets, the ASUS Vivo Tab RT and the Microsoft Surface. I’ve been using the Vivo Tab without a keyboard and relying solely on touch while on the Surface, I’ve been using the cool keyboard/screen cover as well as touch. Overall, I’ve found the devices and Windows 8 very interesting, but still rough around the edges. One example of interesting on the interface formerly known as Metro (IFKaM) is the elongated landscape screens. They take a bit of getting used to, but may prove to be a good choice. On the rough-around-the-edges front, the first thing I currently do on any new device is to install Dropbox so I can get to some files to play around with. Dropbox is not yet available, though it certainly will be—hopefully soon!

IFKaM has a lot to it. On the one hand, it does not seem to me as intuitive as IOS. On the other hand, I have found that if I look hard enough, I can find a way to do something I’m used to doing in Windows. For example, even though I couldn’t use Dropbox, it dawned on me that I could just mount my file server and get to some files that way. As I spend more time with these devices, I’ll have to see if the tradeoff of a longer learning curve pays off in terms of more power and flexibility.

While I ponder that, I want to put in a mention about TouchXPRT 2013 CP1. The source is now available to members. Let us know if you see areas to improve TouchXPRT or the code itself as we work the final version over the next few months. Thanks!

Bill

Comment on this post in the forums

Posted in Touch-based benchmarking, Touch-based computing, Touch-based devices, TouchXPRT |

TouchXPRT in the fast lane

By Bill Catchings

on October 25, 2012

I titled last week’s blog “Putting the TouchXPRT pedal to the metal.” The metaphor still applies. On Monday, we released TouchXPRT 2013 Community Preview 1 (CP1). Members can download it here.

CP1 contains five scenarios based on our research and community feedback. The scenarios are Beautify Photo Album, Prepare Photos for Sharing, Convert Videos for Sharing, Export Podcast to MP3, and Create Slideshow from Photos.

Each scenario gives two types of results. There’s a rate, which allows for simple “bigger is better” comparisons. CP1 also gives the elapsed time for each scenario, which is easier to grasp intuitively. Each approach has its advantages. We’d like to get your feedback on whether you’d like us to pick one of those metrics for the final version of TouchXPRT 2013 or whether it makes more sense to include both. You’ll find a fuller description of the scenarios and the results in the TouchXPRT 2013 Community Preview 1 Design overview.

While you’re looking at CP1, we’re getting the source ready to release. To check out the source, you’ll need a system running Windows 8, with Visual Studio 2012 installed. We hope to release it on Friday. Keep your eye the TouchXPRT forums for more details.

Post your feedback to the TouchXPRT forum, or e-mail it to TouchXPRTSupport@principledtechnologies.com. Do you want more scenarios? Different metrics? A new UI feature? Let us know! Make TouchXPRT the benchmark you want it to be.

As I explained last week, we released CP1 without any restrictions on publishing results. It seems that AnandTech was the first to take advantage of that. Read AnandTech’s Microsoft Surface Review to see TouchXPRT in action.

We are hoping that other folks take advantage of CP1’s capability to act as a cross-platform benchmark on the new class of Windows 8 devices. Come join us in the fast lane!

Bill

Comment on this post in the forums

Posted in Benchmark metrics, Collaborative benchmark development, Touch-based benchmarking, Touch-based devices, TouchXPRT development process, TouchXPRT release cycle, TouchXPRT results |

There is such a thing as too much

TouchXPRT Web test update

Lots of things happening!

Back in September

Touch device first impressions

TouchXPRT in the fast lane

Check out the other XPRTs: