Shortly after the initial
AIXPRT release, we noted that each of the toolkits AIXPRT uses (Intel OpenVINO,
TensorFlow, NVIDIA TensorRT, and Apache MXNet) is on its own development
schedule, and new versions will sometimes appear with little warning. When this
happens, we’ll have to respond by updating specific AIXPRT installation
packages, giving AIXPRT testers relatively short notice.
This is one of those
times! Intel recently released OpenVINO 2020.3 Long-Term Support (LTS), and we’re planning to update the AIXPRT
OpenVINO packages with the LTS version. The LTS version targets environments
that benefit from maximum stability, and don’t require a constant stream of new
tools and feature changes. In other words, it’s well suited for a benchmark,
and we think it’s a good fit for AIXPRT moving forward.
We don’t yet know what
impact the new version will have on AIXPRT OpenVINO test results. A substantial
part of the development process will involve testing the new packages on a
variety of platforms to see how performance changes. We’ll communicate our
findings here in the blog, so AIXPRT testers will know what to expect.
modular nature of the AIXPRT installation packages ensures that we don’t need
to revise the entire AIXPRT suite every time a toolkit update goes live. If you
test with only TensorFlow, TensorRT, or MXNet, or a combination of those
toolkits, this update won’t affect your testing.
We’re not ready to commit
to a release date for the new build, but anticipate it will be in September.
If you have any questions about AIXPRT or OpenVINO, please let us know!
With four separate machine learning toolkits on their own development schedules, three workloads, and a wide range of possible configurations and use cases, AIXPRT has more moving parts than any of the XPRT benchmark tools to date. Because there are so many different components, and because we want AIXPRT to provide consistently relevant evaluation data in the rapidly evolving AI and machine learning spaces, we anticipate a cadence of AIXPRT updates in the future that will be more frequent than the schedules we’ve used for other XPRTs in the past. With that expectation in mind, we want to let AIXPRT testers know that when we release an AIXPRT update, they can expect minimized disruption, consideration for their testing needs, and clear communication.
Each AIXPRT toolkit (Intel OpenVINO, TensorFlow, NVIDIA TensorRT, and Apache MXNet) is on its own development schedule, and we won’t always have a lot of advance notice when new versions are on the way. Hypothetically, a new version of OpenVINO could release one month, and a new version of TensorRT just two months later. Thankfully, the modular nature of AIXPRT’s installation packages ensures that we won’t need to revise the entire AIXPRT suite every time a toolkit update goes live. Instead, we’ll update each package individually when necessary. This means that if you only test with a single AIXPRT package, updates to the other packages won’t affect your testing. For us to maintain AIXPRT’s relevance, there’s unfortunately no way to avoid all disruption, but we’ll work to keep it to a minimum.
Consideration for testers
As we move forward, when software compatibility issues force us to update an AIXPRT package, we may discover that the update has a significant effect on results. If we find that results from the new package are no longer comparable to those from previous tests, we’ll share the differences that we’re seeing in our lab. As always, we will use documentation and versioning to make sure that testers know what to expect and that there’s no confusion about which package to use.
When we update any package, we’ll make sure to communicate any updates in the new build as clearly as possible. We’ll document all changes thoroughly in the package readmes, and we’ll talk through significant updates here in the blog. We’re also available to answer questions about AIXPRT and any other XPRT-related topic, so feel free to ask!