BenchmarkXPRT Blog banner

Tag Archives: TensorFlow

AIXPRT is here!

Justin Greene
By Justin Greene
on November 1, 2019

We’re happy to announce that AIXPRT is now available to the public! AIXPRT includes support for the Intel OpenVINO, TensorFlow, and NVIDIA TensorRT toolkits to run image-classification and object-detection workloads with the ResNet-50 and SSD-MobileNet v1networks, as well as a Wide and Deep recommender system workload with the Apache MXNet toolkit. The test reports FP32, FP16, and INT8 levels of precision.

To access AIXPRT, visit the AIXPRT download page. There, a download table displays the AIXPRT test packages. Locate the operating system and toolkit you wish to test and click the corresponding Download link. For detailed installation instructions and information on hardware and software requirements for each package, click the package’s Readme link. If you’re not sure which AIXPRT package to choose, the AIXPRT package selector tool will help to guide you through the selection process.

In addition, the Helpful Info box on AIXPRT.com contains links to a repository of AIXPRT resources, as well links to XPRT blog discussions about key AIXPRT test configuration settings such as batch size and precision.

We hope AIXPRT will prove to be a valuable tool for you, and we’re thankful for all the input we received during the preview period! If you have any questions about AIXPRT, please let us know.

Posted in AI, AIXPRT, benchmark, BenchmarkXPRT, Cross-platform benchmarks, image classification, Linux, Machine learning, MXNet, NVIDIA, object detection, OpenVINO, Performance benchmarking, recommender system, ResNet-50, SSD-MobileNet v1, TensorFlow, TensorRT, Wide and Deep, Windows | Also tagged , , , , , , , , ,

Coming soon: An interactive AIXPRT selector tool

Justin Greene
By Justin Greene
on October 31, 2019

AI workloads are now relevant to all types of hardware, from servers to laptops to IOT devices, so we intentionally designed AIXPRT to support a wide range of potential hardware, toolkit, and workload configurations. This approach provides AIXPRT testers with a tool that is flexible enough to adapt to a variety of environments. The downside is that the number of options makes it fairly complicated to figure out which AIXPRT download package suits your needs.

To help testers navigate this complexity, we’ve been working on a new interactive selector tool. The tool is not yet live, but the screenshots and descriptions below provide a preview of what’s to come.

The tool will include drop-down menus for the key factors that go into determining the correct AIXPRT download package, along with a description of the options. Users can proceed in any order but will need to make a selection for each category. Since not all combinations work together, each selection the user makes will eliminate some of the options in the remaining categories.

AIXPRT user guide snip 1

After a user selects an option, a check mark appears on the category icon, and the selection for that category appears in the category box (e.g., TensorFlow in the Toolkit category). This shows users which categories they’ve completed and the selections they’ve made. After a user selects options in more than one category, a Start over button appears in the lower-left corner. Clicking this button clears all existing selections and provides users with a clean slate.

Once every category is complete, a Download button appears in the lower-right corner. When you click this, a popup appears that provides a link for the correct download package and associated readme file.

AIXPRT user guide snip 2

We hope the selector tool will help make the AIXPRT download and installation process easier for those who are unfamiliar with the benchmark. Testers who already know exactly which package they need will be able to bypass the tool and go directly to a download table.

The tool will debut with the AIXPRT 1.0 GA in the next few days, and we’ll let everyone know when that happens! If you have any questions or comments about AIXPRT, please let us know.

Justin

Posted in AI, AIXPRT, benchmark, BenchmarkXPRT development community, image classification, IoT, Machine learning, MXNet, object detection, OpenVINO, Performance benchmarking, recommender system, TensorFlow, TensorRT, Wide and Deep | Also tagged , , , , , , ,

AIXPRT Community Preview 3 is here!

Justin Greene
By Justin Greene
on October 3, 2019

We’re happy to announce that the AIXPRT Community Preview 3 (CP3) is now available! As we discussed in last week’s blog, testers can expect three significant changes in AIXPRT CP3:

  • We updated support for the Ubuntu test packages from Ubuntu version 16.04 LTS to version 18.04 LTS.
  • We added TensorRT test packages for Windows and Ubuntu. Previously, AIXPRT testers could test only the TensorFlow variant of TensorRT. Now, they can use TensorRT to test systems with NVIDIA GPUs.
  • We added the Wide and Deep recommender system workload with the MXNet toolkit for Ubuntu systems.


To access AIXPRT CP3, click this access link and submit the brief information form unless you’ve already done so for CP2. You will then gain access to the AIXPRT community preview page. (If you’re not already a BenchmarkXPRT Development Community member, we’ll contact you with more information about your membership.)

On the community preview page, a download table displays the currently available AIXPRT CP3 test packages. Locate the operating system and toolkit you wish to test, and click the corresponding Download link. For detailed installation instructions and information on hardware and software requirements for each package, click the corresponding Readme link. Instead of providing installation guide PDFs as we did for CP2, we are now directing testers to a public GitHub repository. The repository contains the installation readmes for all the test packages, as well as a selection of alternative test configuration files. We’ll discuss the alternative configuration files in more detail in a future blog post.

Note: Those who have access to the existing AIXPRT GitHub repository will be able to access CP3 in the same way as previous versions.

We’ll continue to keep everyone up to date with AIXPRT news here in the blog. If you have any questions or comments, please let us know.

Justin

Posted in AI, AIXPRT, benchmark, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Future of performance evaluation, image classification, Machine learning, MXNet, NVIDIA, object detection, OpenVINO, Performance benchmarking, recommender system, ResNet-50, SSD-MobileNet v1, TensorFlow, TensorRT, Ubuntu, Wide and Deep | Also tagged , , , , , , , , , , , ,

Understanding concurrent instances in AIXPRT

Justin Greene
By Justin Greene
on September 12, 2019

Over the past few weeks, we’ve discussed several of the key configuration variables in AIXPRT, such as batch size and level of precision. Today, we’re discussing another key variable: number of concurrent instances. In the context of machine learning inference, this refers to how many instances of the network model (ResNet-50, SSD-MobileNet, etc.) the benchmark runs simultaneously.

By default, the toolkits in AIXPRT run one instance at a time and distribute the compute load according to the characteristics of the CPU or GPU under test, as well as any relevant optimizations or accelerators in the toolkit’s reference library. By setting the number of concurrent instances to a number greater than one, a tester can use multiple CPUs or GPUs to run multiple instances of a model at the same time, usually to increase throughput.

With multiple concurrent instances, a tester can leverage additional compute resources to potentially achieve higher throughput without sacrificing latency goals.

In the current version of AIXPRT, testers can run multiple concurrent instances in the OpenVINO, TensorFlow, and TensorRT toolkits. When AIXPRT Community Preview 3 becomes available, this option will extend to the MXNet toolkit. OpenVINO and TensorRT automatically allocate hardware for each instance and don’t let users make manual adjustments. TensorFlow and MXNet require users to manually bind instances to specific hardware. (Manual hardware allocation for multiple instances is more complicated than we can cover today, so we may devote a future blog entry to that topic.)

Setting the number of concurrent instances in AIXPRT

The screenshot below shows part of a sample config file (the same one we used when we discussed batch size and precision). The value in the “concurrent instances” row indicates how many concurrent instances will be operating during the test. In this example, the number is one. To change that value, a tester simply replaces it with the desired number and saves the changes.

Config_snip

If you have any questions or comments (about concurrent instances or anything else), please feel free to contact us.

Justin

Posted in AI, AIXPRT, Benchmarking, Benchmarking computing devices, Community Preview, Cross-platform benchmarks, Machine learning, MXNet, OpenVINO, Performance benchmarking, ResNet-50, SSD-MobileNet v1, TensorFlow, TensorRT | Also tagged , , , , , , , , ,

Navigating the AIXPRT Community Preview download page just got easier

Justin Greene
By Justin Greene
on July 25, 2019

AIXPRT Community Preview 2 (CP2) has been generating quite a bit of interest among the BenchmarkXPRT Development Community and members of the tech press. We’re excited that the tool has piqued curiosity and that folks are recognizing its value for technical analysis. When talking with folks about test setup and configuration, we keep hearing the same questions:

  • How do I find the exact toolkit or package that I need?
  • How do I find the instructions for a specific toolkit?
  • What test configuration variables are most important for producing consistent, relevant results?
  • How do I know which values to choose when configuring options such as iterations, concurrent instances, and batch size?


In the coming weeks, we’ll be working to provide detailed answers to questions about test configuration. In response to the confusion about finding specific packages and instructions, we’ve redesigned the CP2 download page to make it easier for you to find what you need. Below, we show a snapshot from the new CP2 download table. Instead of having to download the entire CP2 package that includes the OpenVINO, TensorFlow, and TensorRT in TensorFlow test packages, you can now download one package at a time. In the Documentation column, we’ve posted package-specific instructions, so you won’t have to wade through the entire installation guide to find the instructions you need.

AIXPRT Community Preview download table

We hope these changes make it easier for people to experiment with AIXPRT. As always, please feel free to contact us with any questions or comments you may have.

Justin

Posted in AI, AIXPRT, AMD, benchmark, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Intel, Linux, NVIDIA, OpenVINO, TensorFlow, TensorRT, Windows 10 | Also tagged , , , , , , , ,

Making AIXPRT easier to use

Justin Greene
By Justin Greene
on June 20, 2019

We’re glad to see so much interest in the AIXPRT CP2 build. Over the past few days, we’ve received two questions about the setup process: 1) where to find instructions for setting up AIXPRT on Windows, and 2) whether we could make it easier to install Intel OpenVINO on test systems.

In response to the first question, testers can find the relevant instructions for each framework in the readme files included in the AIXPRT install package. Instructions for Windows installation are in section 3 of the OpenVINO and TensorFlow readmes. Please note that whether you’re running AIXPRT on Ubuntu or Windows, be sure to read the “Known Issues” section in the readme, as there may be issues relevant to your specific configuration.

The readme files for each respective framework in the CP2 package are located here:

  • AIXPRT_0.5_CP2\AIXPRT_OpenVINO_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning
  • AIXPRT_0.5_CP2\AIXPRT_TensorFLow_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning
  • AIXPRT_0.5_CP2\AIXPRT_TensorFlow_TensorRT_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning


We’re also working on consolidating the instructions into a central document that will make it easier for everyone to find the instructions they need.

In response to the question about OpenVINO installation, we’re working on an AIXPRT CP2 package that includes a precompiled version of OpenVINO R5.0.1 for easy installation on Windows via a few quick commands, and a script that installs the necessary OpenVINO dependencies. We’re currently testing the build, and we’ll make it available to testers as soon as possible.

The tests themselves will not change, so the new build will not influence existing results from Ubuntu or Windows. We hope it will simply facilitate the setup and testing process for many users.

We appreciate each bit of feedback that we receive, so if you have any suggestions for AIXPRT, please let us know!

Justin

Posted in AI, AIXPRT, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Linux, Machine learning, OpenVINO, TensorFlow, TensorRT, Ubuntu, What makes a good benchmark?, Windows 10 | Also tagged , , , , , ,

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x