PT-Logo
Forgot your password?
BenchmarkXPRT Blog banner

Category: Open Source

Principled Technologies and the BenchmarkXPRT Development Community make the AIXPRT source code available to the public

Durham, NC, February 18 — Principled Technologies and the BenchmarkXPRT Development Community release the source code for the AIXPRT benchmark to the public. AIXPRT is a free tool that allows users to evaluate a system’s machine learning inference performance by running common image-classification, object detection, and recommender system workloads.

“Publishing the AIXPRT source code is part of our commitment to making the XPRT development process as transparent as possible,” said Bill Catchings, co-founder of Principled Technologies, which administers the BenchmarkXPRT Development Community. “By allowing all interested parties to download and review our source code, we’re taking tangible steps to improve openness in the benchmarking industry.”

To access the AIXPRT source code, visit the AIXPRT GitHub repository at https://github.com/BenchmarkXPRT/AIXPRT.

AIXPRT includes support for the Intel© OpenVINO™, TensorFlow™, and NVIDIA© TensorRT™ toolkits to run image-classification and object-detection workloads with the ResNet-50 and SSD-MobileNet v1 networks, as well as the MXNet™ toolkit with a Wide and Deep recommender system workload. The test reports FP32, FP16, and INT8 levels of precision.

To access AIXPRT, visit www.AIXPRT.com.

AIXPRT is part of the BenchmarkXPRT suite of performance evaluation tools, which includes WebXPRT, CrXPRT, MobileXPRT, TouchXPRT, and HDXPRT. The XPRTs help users get the facts before they buy, use, or evaluate tech products such as computers, tablets, and phones.

To learn more about the BenchmarkXPRT Development Community, go to www.BenchmarkXPRT.com or contact a BenchmarkXPRT Development Community representative directly by sending a message to BenchmarkXPRTsupport@PrincipledTechnologies.com.

About Principled Technologies, Inc.
Principled Technologies, Inc. is a leading provider of technology marketing, as well as learning and development services. It administers the BenchmarkXPRT Development Community.

Principled Technologies, Inc. is located in Durham, North Carolina, USA. For more information, please visit www.PrincipledTechnologies.com.

Company Contact
Justin Greene
BenchmarkXPRT Development Community
Principled Technologies, Inc.
1007 Slater Road, Ste. 300
Durham, NC 27704
BenchmarkXPRTsupport@PrincipledTechnologies.com

The AIXPRT source code is now public

This week, we have good news for AIXPRT testers: the AIXPRT source code is now available to the public via GitHub. As we’ve discussed in the past, publishing XPRT source code is part of our commitment to making the XPRT development process as transparent as possible. With other XPRT benchmarks, we’ve only made the source code available to community members. With AIXPRT, we have released the source code more widely. By allowing all interested parties, not just community members, to download and review our source code, we’re taking tangible steps to improve openness and honesty in the benchmarking industry and we’re encouraging the kind of constructive feedback that helps to ensure that the XPRTs continue to contribute to a level playing field.

Traditional open-source models encourage developers to change products and even take them in new and different directions. Because benchmarking requires a product that remains static to enable valid comparisons over time, we allow people to download the source code and submit potential workloads for future consideration, but we reserve the right to control derivative works. This discourages a situation where someone publishes an unauthorized version of the benchmark and calls it an “XPRT.”

We encourage you to download and review the source and send us any feedback you may have. Your questions and suggestions may influence future versions of AIXPRT. If you have any questions about AIXPRT or accessing the source code, please feel free to ask! Please also let us know if you think we should take this approach to releasing the source code with other XPRT benchmarks.

Justin

Comparing open source and open development

Why do we use open development when designing and building the XPRTs, and what’s the difference between our open development approach and traditional open-source methods? The terminology around these two models can be confusing, so we wanted to review some similarities and differences.

Why open development?

An open development approach helps encourage collaboration, innovation, and transparency. XPRT community members get involved in the development of each benchmark from the beginning:

  • They submit suggestions, questions, and concerns that inform the future design of the tools.
  • They view early proposals for new versions and contribute comments for the final design.
  • They suggest new workloads.
  • They have access to community previews (beta builds) of the tools.
  • They submit source code for inclusion in the benchmarks.
  • They examine existing source code.

A commitment to transparency

Because we’re committed to publishing reliable, unbiased benchmarks, we also want make the XPRT development process as transparent as possible. It’s not unusual for people to claim that any given benchmark contains hidden biases. To address this problem, we make our source code available to anyone who joins the community. This approach reduces the risk of unforeseen bias in our benchmarks.

Quality control

Unlike open-source models, open development allows us to control derivative works, which can be important in benchmarking. While open source encourages a constantly evolving product that may fork into substantially different versions, benchmarking requires a product that remains static to enable valid comparisons over time. By controlling derivative works, we can avoid the problem of unauthorized versions of the benchmarks being published as “XPRTs.”

In the future, we may use a traditional open-source model for specific XPRTs or other projects. If we do, we’ll share our reasoning with the community and ask for their thoughts about the best way to proceed. If you’re not a community member, but are interested in benchmark development, we encourage you to join today!

Justin

Keeping up with the latest Android news

Ars Technica recently published a deep-dive review of Android 8.0 (Oreo) that contains several interesting tidbits about what the author called “Android’s biggest re-architecture, ever.” After reading the details, it’s hard to argue with that assessment.

The article’s thorough analysis includes a list of the changes Oreo is bringing to the UI, notification settings, locations service settings, and more. In addition to the types of updates that we usually see, a few key points stand out.

  • Project Treble, a complete reworking of Android’s foundational structure intended to increase the speed and efficiency of update delivery
  • A serious commitment to eliminating silent background services, giving users more control over their phone’s resources, and potentially enabling significant gains in battery life
  • Increased machine learning/neural network integration for text selection and recognition
  • A potential neural network API that allows third-party plugins
  • Android Go, a scaled-down version of Android tuned for budget phones in developing markets


There’s too much information about each of the points to discuss here, but I encourage anyone interested in Android development to check out the article. Just be warned that when they say “thorough,” they mean it, so it’s not exactly a quick read.

Right now, Oreo is available on only the Google Pixel and Pixel XL phones, and will not likely be available to most users until sometime next year. Even though widespread adoption is a way off, the sheer scale of the expected changes requires us to adopt a long-term development perspective.

We’ll continue to track developments in the Android world and keep the community informed about any impact that those changes may have on MobileXPRT and BatteryXPRT. If you have any questions or suggestions for future XPRT/Android applications, let us know!

Justin

Looking under the hood

In the next couple of weeks, we’ll publish the source code and build instructions for the latest HDXPRT 2014 and BatteryXPRT 2014 builds. Access to XPRT source code is one of the benefits of BenchmarkXPRT Development Community membership. For readers who may not know, this a good time to revisit the reasons we make the source code available.

The primary reason is transparency; we want the XPRTs to be as open as possible. As part of our community model for software development, the source code is available to anyone who joins the community. Closed-source benchmark development can lead some people to infer that a benchmark is biased in some way. Our approach makes it impossible to hide any biases.

Another reason we publish source code is to encourage collaborative development and innovation. Community members are involved in XPRT development from the beginning, helping to identify emerging technologies in need of reliable benchmarking tools, suggesting potential workloads and improvements, reviewing design documents, and offering all sorts of general feedback.

Simply put, if you’re interested in benchmarking and the BenchmarkXPRT Development Community, then we’re interested in what you have to say! Community input helps us at every step of the process, and ultimately helps us to create benchmarking tools that are as reliable and relevant as possible.

If you’d like to review XPRT source code, but haven’t yet joined the community, we encourage you to go ahead and join! It’s easy, and if you work for a company or organization with an interest in benchmarking, you can join the community for free. Simply fill out the form with your company e-mail address and click the option to be considered for a free membership. We’ll contact you to verify the address is real and then activate your membership.

If you have any other questions about community membership or XPRT source code, feel free to contact us. We look forward to hearing from you!

Justin

Open source?

We’re proud of the BenchmarkXPRT Development Community and its accomplishments over the last five years. We’re also thankful for the contributions the members of the community have made. One of the benefits of membership is access to the source code for all the XPRT performance tools. This has meant that the code is available to anyone willing to take the easy step of joining the community.

Behind our decision to use this model rather than a more traditional, open-source model was the need to control derivative works. The license agreement for the source allows members to modify the source, but not to claim that the results from that derivative code are XPRT results. For example, as a member, you may download the TouchXPRT source and modify the workloads for your specific purposes, but you can’t refer to the results as TouchXPRT results.

After much thought and discussion, we have come to believe that we can protect the benchmarks’ reputation within a traditional, open-source framework. While our original concerns are still valid, we think that the success and stature of the XPRTs is such that we can make it available via open source.

However, before we take this step, we want to hear the thoughts, concerns, and opinions of both our community members and the wider public.

Please note that if we do make the code open source, the other benefits of being a member—access to requests for comment, design documents, and community previews—will not change.

Please let us know that you think. Email us or contact us on Twitter.

Bill

Check out the other XPRTs: