Recently, a member of the tech
press asked us about the status of AIXPRT,
our benchmark that measures machine learning inference performance. We want to
share our answer here in the blog for the benefit of other readers. The writer said
it seemed like we had not updated AIXPRT in a long time, and wondered whether we
had any immediate plans to do so.
true that we haven’t updated AIXPRT in quite some time. Unfortunately, while a
few tech press publications and OEM labs began experimenting with AIXPRT
testing, the benchmark never got the traction we hoped for, and we’ve decided
to invest our resources elsewhere for the time being. The AIXPRT installation
packages are still available for people to use or reference as they wish, but
we have not updated the benchmark to work with the latest platform versions
(OpenVINO, TensorFlow, etc.). It’s likely that several components in each
package are out of date.
are interested in AIXPRT and would like us to bring it up to date, please let us know.
We can’t promise that we’ll revive the benchmark, but your feedback could be a
valuable contribution as we try to gauge the benchmarking community’s interest.
Earlier this month, we
discussed a few of our ideas for possible changes in WebXPRT 4, including
new web technologies that may work well in a browser benchmark. Today, we’re
going to focus on one of those technologies, WebAssembly, in more detail.
(WASM) is a binary instruction format that works across all modern browsers. WASM
provides a sandboxed environment that operates at native speeds and takes
advantage of common hardware specs across platforms. WASM’s capabilities offer
web developers a great deal of flexibility for running complex client
applications in the browser. That level of flexibility may enable workload scenario
options for WebXPRT 4 such as
gaming, video editing, VR, virtual machines, and image recognition. We’re
excited about those possibilities, but it remains to be seen which WASM use
cases will meet the criteria we look for when considering new WebXPRT
workloads, such as relevancy to real life, consistency and replicability, and
the broadest-possible level of cross-browser support.
One WASM workload that
we’re investigating is a web-based machine learning workload with TensorFlow
tasks, including image classification, object detection, sentence encoding, and
natural language processing. TensorFlow.js originally used WebGL technology on
the back end, but now it’s possible to run the workload using WASM. We could
also use this technology to enhance one of WebXPRT’s existing AI-themed
workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.
We’re can’t yet say that a WASM workload will definitely appear in WebXPRT 4, but the technology is promising. Do you have any experience with WASM, or ideas for WASM workloads? There’s still time for you to help shape the future of WebXPRT 4, so let us know what you think!
We’re happy to
announce that the AIXPRT learning tool is now live! We
designed the tool to serve as an information hub for common AIXPRT topics and
questions, and to help tech journalists, OEM lab engineers, and everyone who is
interested in AIXPRT find the answers they need in as little time as possible.
The tool features four
primary areas of content:
The Q&A section provides quick answers to the questions we
receive most from testers and the tech press.
The AIXPRT: the basics section describes specific topics such as
the benchmark’s toolkits, networks, workloads, and hardware and software
The testing and results section covers the testing process,
metrics, and how to publish results.
The AI/ML primer provides brief, easy-to-understand definitions of
key AI and ML terms and concepts for those who want to learn more about the
The first screenshot below shows the home screen. To show how some of the popup information sections appear, the second screenshot shows the Inference tasks (workloads) entry in the AI/ML Primer section.
We’re excited about the new AIXPRT learning tool, and we’re also happy to report that we’re working on a version of the tool for CloudXPRT. We hope to make the CloudXPRT tool available early next year, and we’ll post more information in the blog as we get closer to taking it live.
If you have any questions about the tool, please let us know!
Last month, we announced that we’re working on
a new AIXPRT learning tool. Because we want tech journalists, OEM lab
engineers, and everyone who is interested in AIXPRT to be able to find the
answers they need in as little time as possible, we’re designing this tool to serve
as an information hub for common AIXPRT topics and questions.
We’re still finalizing
aspects of the tool’s content and design, so some details may change, but we
can now share a sneak peak of the main landing page. In the screenshot below,
you can see that the tool will feature four primary areas of content:
The FAQ section will provide quick answers to the questions we
receive most from testers and the tech press.
The AIXPRT basics section will describe specific topics such as the
benchmark’s toolkits, networks, workloads, and hardware and software
The testing and results section will cover the testing process,
the metrics the benchmark produces, and how to publish results.
The AI/ML primer will provide brief, easy-to-understand definitions
of key AI and ML terms and concepts for those who want to learn more about the
We’re excited about the new AIXPRT learning tool, and will share more information here in the blog as we get closer to a release date. If you have any questions about the tool, please let us know!
This week, we’re sharing news on two topics that we’ve discussed
here in the blog over the past several months: CloudXPRT v1.01 and a potential
AIXPRT OpenVINO update.
Last week, we announced that we were very close to releasing an
updated CloudXPRT build (v1.01) with two minor bug fixes, an improved post-test
results processing script, and an adjustment to one of our test configuration
recommendations. Our testing and prep is complete, and the new version is live
in the CloudXPRT GitHub repository and on our site!
None of the v1.01
changes affect performance or test results, so scores from the new build are
comparable to those from previous CloudXPRT builds. If you’d like to know more
about the changes, take a look at last week’s blog post.
The AIXPRT OpenVINO
In late July, we discussed our plans to update the AIXPRT OpenVINO packages
with OpenVINO 2020.3 Long-Term Support (LTS). While there are no
known problems with the existing AIXPRT OpenVINO package, the LTS version
targets environments that benefit from maximum stability and don’t require a
constant stream of new tools and feature changes, so we thought it would be
well suited for a benchmark like AIXPRT.
We initially believed that
the update process would be relatively simple, and we’d be able to release a
new AIXPRT OpenVINO package in September. However, we’ve discovered that the
process is involved enough to require substantial low-level recoding. At this
time, it’s difficult to estimate when the updated build will be ready for
release. For any testers looking forward to the update, we apologize for the
If you have any questions or comments about
these or any other XPRT-related topics, please let us know!
For anyone interested
in learning more about AIXPRT, the Introduction to AIXPRT white paper provides detailed information
about its toolkits, workloads, system requirements, installation, test
parameters, and results. However, for AIXPRT.com visitors who want to find the answers to specific
AIXPRT-related questions quickly, a white paper can be daunting.
Because we want tech
journalists, OEM lab engineers, and everyone who is interested in AIXPRT to be
able to find the answers they need in as little time as possible, we’ve decided
to develop a new learning tool that will serve as an information hub for common
AIXPRT topics and questions.
The new learning tool
will be available online through our site. It will offer quick bites of
information about the fundamentals of AIXPRT, why the benchmark matters, the
benefits of AIXPRT testing and results, machine learning concepts, key terms,
and practical testing concerns.
We’re still working on the tool’s content and design. Because we’re designing this tool for you, we’d love to hear the topics and questions you think we should include. If you have any suggestions, please let us know!