BenchmarkXPRT Blog banner

Month: April 2018

The WebXPRT 3 source code is now available

We’re excited to announce that the WebXPRT 3 source code is now available to BenchmarkXPRT Development Community members!

Download the WebXPRT 3 source here (login required).

We’ve also posted a download link on the WebXPRT tab in the Members’ Area. The source code package contains instructions for setting up and configuring a local instance of WebXPRT for those who wish to do so.

As part of our community model for software development, source code for each of the XPRTs is available to anyone who joins the community. If you’d like to review XPRT source code, but haven’t yet joined the community, we encourage you to join! Registration is quick and easy, and if you work for a company or organization with an interest in benchmarking, you can join the community for free. Simply fill out the form with your company e-mail address and select the option to be considered for a free membership. We’ll contact you to verify the address and then activate your membership.

If you have any other questions about community membership or XPRT source code, feel free to contact us. We look forward to hearing from you!

Justin

How the XPRTs handle your data

Data privacy is a hot topic in the news these days. In an ideal world, all applications and websites that have access to users’ sensitive personal information would treat that information with respect. Users could be confident that their data would not be abused, secretly transmitted to third parties, or used as a launchpad for extensive violations of their privacy.

In the real world, the situation is often quite different, but not with the XPRTs. Just as we strive for transparency during the benchmark development process, we try to be completely upfront regarding how we handle your personal data. We’re committed to the principle that your personal information belongs to you and no one else. We don’t gather, store, or disseminate any of your data without your knowledge and consent, and we never try to trick you with misleading terms or pages of legal jargon that few will ever read. We take that commitment very seriously.

To join the BenchmarkXPRT Development Community, you need to provide only your first and last name, corporate affiliation, and valid email address. These pieces of information form your profile, which other members of the community can view only if you choose to participate in the BenchmarkXPRT forum. We will not share your profile information, or even the fact that you are a member, with anyone outside the community. If you do not participate in the forum, even other community members will not know you are a member.

For the XPRT apps, we gather data only for the purposes of ensuring quality and improving the benchmark. Two apps—WebXPRT and CrXPRT—collect test results and some data about the browser and device that produced those results. While we may refer to high and low scores or averages in the materials we publish, we will never make an individual WebXPRT result public unless the tester requests we do so. These apps report no identifying personal or corporate data, or any other potentially confidential information.

None of the remaining XPRT apps—BatteryXPRT, HDXPRT, MobileXPRT, and TouchXPRT—collect any data. When an individual runs one of these apps, we see no information or results unless they submit a result for publication or send us a direct message mentioning the result. This makes the XPRTs ideal for pre-production OEM testing because there’s no risk of model or performance information leaking to the press after automatically appearing on a benchmark’s website.

You can read more about how we handle your data on our privacy policy page. If you have any questions, please feel free to ask!

Justin

AIXPRT: We want your feedback!

Today, we’re publishing the AIXPRT Request for Comments (RFC) document. The RFC explains the need for a new artificial intelligence (AI)/machine learning benchmark, shows how the BenchmarkXPRT Development Community plans to address that need, and provides preliminary design specifications for the benchmark.

We’re seeking feedback and suggestions from anyone interested in shaping the future of machine learning benchmarking, including those not currently part of the Development Community. Usually, only members of the BenchmarkXPRT Development Community have access to our RFCs and the opportunity to provide feedback. However, because we’re seeking input from non-members who have expertise in this field, we will be posting this RFC in the New events & happenings section of the main BenchmarkXPRT.com page and making it available at AIXPRT.com.

We welcome input on all aspects of the benchmark, including scope, workloads, metrics and scores, UI design, and reporting requirements. We will accept feedback through May 13, 2018, after which BenchmarkXPRT Development Community administrators will collect and evaluate the feedback and publish the final design specification.

Please share the RFC with anyone interested in machine learning benchmarking and please send us your feedback before May 13.

Justin

Comparing open source and open development

Why do we use open development when designing and building the XPRTs, and what’s the difference between our open development approach and traditional open-source methods? The terminology around these two models can be confusing, so we wanted to review some similarities and differences.

Why open development?

An open development approach helps encourage collaboration, innovation, and transparency. XPRT community members get involved in the development of each benchmark from the beginning:

  • They submit suggestions, questions, and concerns that inform the future design of the tools.
  • They view early proposals for new versions and contribute comments for the final design.
  • They suggest new workloads.
  • They have access to community previews (beta builds) of the tools.
  • They submit source code for inclusion in the benchmarks.
  • They examine existing source code.

A commitment to transparency

Because we’re committed to publishing reliable, unbiased benchmarks, we also want make the XPRT development process as transparent as possible. It’s not unusual for people to claim that any given benchmark contains hidden biases. To address this problem, we make our source code available to anyone who joins the community. This approach reduces the risk of unforeseen bias in our benchmarks.

Quality control

Unlike open-source models, open development allows us to control derivative works, which can be important in benchmarking. While open source encourages a constantly evolving product that may fork into substantially different versions, benchmarking requires a product that remains static to enable valid comparisons over time. By controlling derivative works, we can avoid the problem of unauthorized versions of the benchmarks being published as “XPRTs.”

In the future, we may use a traditional open-source model for specific XPRTs or other projects. If we do, we’ll share our reasoning with the community and ask for their thoughts about the best way to proceed. If you’re not a community member, but are interested in benchmark development, we encourage you to join today!

Justin

Check out the other XPRTs:

Forgot your password?