BenchmarkXPRT Blog banner

Author Archives: Justin Greene

An update on HDXPRT development

It’s been a while since we updated the community on HDXPRT development, and we’ve made a lot of progress since then. Here’s a quick summary of where we are and what to expect in the coming months.

The benchmark’s official name will be HDXPRT 4, and we’re sticking with the basic plan we outlined in the blog, which includes updating the benchmark’s real-world trial applications and workload content and improving the UI.

We’ve updated Adobe Photoshop Elements, Audacity, CyberLink Media Espresso, and HandBrake to more contemporary versions, but decided the benchmark will no longer use Apple iTunes. We sometimes encountered problems with iTunes during testing, and because we can complete the audio-related workloads using Audacity, we decided that it was OK to remove iTunes from the test. Please contact us if you have any concerns about this decision.

In addition to the editing photos, editing music, and converting videos workloads from prior versions of the benchmark, HDXPRT 4 includes two new Photoshop Elements scenarios. The first utilizes an AI tool that corrects closed eyes in photos and the second creates a single panoramic photo from seven separate photos. For the photo and video workloads, we produced new high-res photo content and 4K GoPro video footage respectively.

For the UI, our goal is to implement a clean and functional design and align it more closely with the themes, colors, and font styles we’ll be implementing in the XPRTs moving forward. The WebXPRT 3 UI will give you a feel for the direction the HDXPRT UI is headed.

Some of these details may change as we test preliminary builds, but we wanted to give you a better sense of where HDXPRT is headed. We’re not ready to share a date for the community preview, but will provide more details as the day approaches.

If you have any questions or comments about HDXPRT, please let us know. It’s not too late to for us to consider your input for HDXPRT 4.

Justin

The WebXPRT 3 source code is now available

We’re excited to announce that the WebXPRT 3 source code is now available to BenchmarkXPRT Development Community members!

Download the WebXPRT 3 source here (login required).

We’ve also posted a download link on the WebXPRT tab in the Members’ Area. The source code package contains instructions for setting up and configuring a local instance of WebXPRT for those who wish to do so.

As part of our community model for software development, source code for each of the XPRTs is available to anyone who joins the community. If you’d like to review XPRT source code, but haven’t yet joined the community, we encourage you to join! Registration is quick and easy, and if you work for a company or organization with an interest in benchmarking, you can join the community for free. Simply fill out the form with your company e-mail address and select the option to be considered for a free membership. We’ll contact you to verify the address and then activate your membership.

If you have any other questions about community membership or XPRT source code, feel free to contact us. We look forward to hearing from you!

Justin

How the XPRTs handle your data

Data privacy is a hot topic in the news these days. In an ideal world, all applications and websites that have access to users’ sensitive personal information would treat that information with respect. Users could be confident that their data would not be abused, secretly transmitted to third parties, or used as a launchpad for extensive violations of their privacy.

In the real world, the situation is often quite different, but not with the XPRTs. Just as we strive for transparency during the benchmark development process, we try to be completely upfront regarding how we handle your personal data. We’re committed to the principle that your personal information belongs to you and no one else. We don’t gather, store, or disseminate any of your data without your knowledge and consent, and we never try to trick you with misleading terms or pages of legal jargon that few will ever read. We take that commitment very seriously.

To join the BenchmarkXPRT Development Community, you need to provide only your first and last name, corporate affiliation, and valid email address. These pieces of information form your profile, which other members of the community can view only if you choose to participate in the BenchmarkXPRT forum. We will not share your profile information, or even the fact that you are a member, with anyone outside the community. If you do not participate in the forum, even other community members will not know you are a member.

For the XPRT apps, we gather data only for the purposes of ensuring quality and improving the benchmark. Two apps—WebXPRT and CrXPRT—collect test results and some data about the browser and device that produced those results. While we may refer to high and low scores or averages in the materials we publish, we will never make an individual WebXPRT result public unless the tester requests we do so. These apps report no identifying personal or corporate data, or any other potentially confidential information.

None of the remaining XPRT apps—BatteryXPRT, HDXPRT, MobileXPRT, and TouchXPRT—collect any data. When an individual runs one of these apps, we see no information or results unless they submit a result for publication or send us a direct message mentioning the result. This makes the XPRTs ideal for pre-production OEM testing because there’s no risk of model or performance information leaking to the press after automatically appearing on a benchmark’s website.

You can read more about how we handle your data on our privacy policy page. If you have any questions, please feel free to ask!

Justin

AIXPRT: We want your feedback!

Today, we’re publishing the AIXPRT Request for Comments (RFC) document. The RFC explains the need for a new artificial intelligence (AI)/machine learning benchmark, shows how the BenchmarkXPRT Development Community plans to address that need, and provides preliminary design specifications for the benchmark.

We’re seeking feedback and suggestions from anyone interested in shaping the future of machine learning benchmarking, including those not currently part of the Development Community. Usually, only members of the BenchmarkXPRT Development Community have access to our RFCs and the opportunity to provide feedback. However, because we’re seeking input from non-members who have expertise in this field, we will be posting this RFC in the New events & happenings section of the main BenchmarkXPRT.com page and making it available at AIXPRT.com.

We welcome input on all aspects of the benchmark, including scope, workloads, metrics and scores, UI design, and reporting requirements. We will accept feedback through May 13, 2018, after which BenchmarkXPRT Development Community administrators will collect and evaluate the feedback and publish the final design specification.

Please share the RFC with anyone interested in machine learning benchmarking and please send us your feedback before May 13.

Justin

Comparing open source and open development

Why do we use open development when designing and building the XPRTs, and what’s the difference between our open development approach and traditional open-source methods? The terminology around these two models can be confusing, so we wanted to review some similarities and differences.

Why open development?

An open development approach helps encourage collaboration, innovation, and transparency. XPRT community members get involved in the development of each benchmark from the beginning:

  • They submit suggestions, questions, and concerns that inform the future design of the tools.
  • They view early proposals for new versions and contribute comments for the final design.
  • They suggest new workloads.
  • They have access to community previews (beta builds) of the tools.
  • They submit source code for inclusion in the benchmarks.
  • They examine existing source code.

A commitment to transparency

Because we’re committed to publishing reliable, unbiased benchmarks, we also want make the XPRT development process as transparent as possible. It’s not unusual for people to claim that any given benchmark contains hidden biases. To address this problem, we make our source code available to anyone who joins the community. This approach reduces the risk of unforeseen bias in our benchmarks.

Quality control

Unlike open-source models, open development allows us to control derivative works, which can be important in benchmarking. While open source encourages a constantly evolving product that may fork into substantially different versions, benchmarking requires a product that remains static to enable valid comparisons over time. By controlling derivative works, we can avoid the problem of unauthorized versions of the benchmarks being published as “XPRTs.”

In the future, we may use a traditional open-source model for specific XPRTs or other projects. If we do, we’ll share our reasoning with the community and ask for their thoughts about the best way to proceed. If you’re not a community member, but are interested in benchmark development, we encourage you to join today!

Justin

New opportunities for TouchXPRT

Next week’s XPRT Weekly Tech Spotlight will feature a unique device: the HP Envy x2 2-in-1. The first device of its kind on the market, the Envy x2 runs Windows 10 on ARM hardware—in this case, a Qualcomm Snapdragon 835 platform. ASUS and Lenovo will release similar devices in the coming months. Using the ARM chips found in many flagship phones, these devices aim to power robust operating systems on 2-in-1s and laptops while providing extended battery life and always-on LTE connections.

These new devices bring ample opportunities for benchmarking. Consumers will want to know about potential trade-offs between price, power, and battery life—incentivizing tech reviewers to dive into the details and provide useful data points. But for the new Windows on ARM systems, the usual benchmarks have presented challenges. Many traditional laptop benchmarks just won’t work on the new systems. TouchXPRT, however, works like a charm.

TouchXPRT assesses performance on any Windows device. Since it’s a Universal Windows Platform (UWP) app that runs on both x86 and ARM systems, it can evaluate how well a Windows device running on ARM hardware performs compared to traditional laptops and 2-in-1s. It’s easy to install, takes about 15 minutes to run, and you can download it directly from TouchXPRT.com or install it from the Microsoft Store. Labs can also automate testing using the command line or a script.

If you’ve been looking for a Windows performance evaluation tool that’s easy to use and has the flexibility of a UWP app, give TouchXPRT a try. Read more details about TouchXPRT here, and please don’t hesitate to contact us with any questions you may have.

Justin

Check out the other XPRTs:

Forgot your password?