OpenVINO

OpenVINO is a free toolkit designed to make it easier to deploy AI-based computer vision applications. AIXPRT supports the Intel distribution of OpenVINO to run image-classification and object-detection workloads with the ResNet-50 and SSD-MobileNet v1 networks. Learn more at https://docs.openvinotoolkit.org/.

TensorFlow

TensorFlow, developed by Google, is a free, open source machine learning toolkit. AIXPRT supports TensorFlow to run image-classification and object-detection workloads with the ResNet-50 and SSD-MobileNet v1 networks. Learn more at https://www.tensorflow.org/.

TensorRT

NVIDIA TensorRT is a free software development kit for deep learning inference applications. AIXPRT supports TensorRT to run image classification and object-detection workloads with the ResNet-50 and SSD-MobileNet v1 networks. Learn more at https://developer.nvidia.com/tensorrt.

MXNet

Apache MXNet is a free, open source deep learning toolkit for deep learning inference applications. AIXPRT supports MXNet to run a recommender system workload with a Wide and Deep network. Learn more at https://mxnet.apache.org/.

ResNet-50

ResNet-50 is a convolutional neural network designed for image classification tasks. Image classification, or recognition, is the process by which a trained convolutional neural network model analyzes an image and predicts (infers) what the image represents. Learn more at https://www.mathworks.com/help/deeplearning/ref/resnet50.html.

SSD-MobileNet

SSD-MobileNet is a Single-Shot Multibox Detector (SSD) network designed for object detection tasks. Object detection is the process by which a trained convolutional neural network model analyzes a still or video image and identifies the presence and location of multiple objects. Learn more at https://www.mathworks.com/help/deeplearning/ug/object-detection-using-ssd-deep-learning.html.

Wide and Deep

Wide and Deep learning networks combine the memorization efficiency of linear learning (wide) models with the generalized inference powers of neural network (deep) models to complete recommender system tasks. Recommender systems are ubiquitous in modern life, and serve as the brains behind the suggestion and auto-complete features we see in streaming content services, social media, navigation, and targeted advertising.

Learn more at https://arxiv.org/abs/1606.07792.

Image classification

Image classification, or recognition, is the process by which a trained convolutional neural network model analyzes an image and predicts (infers) what the image represents. For example, during training, the model may analyze thousands of images depicting three types of big cats: cheetahs, lions, and tigers. The model learns to associate specific properties of those images with each of those three labels. Later, when a workload task presents the model with an inference request by asking for classification of a picture of a big cat, the model outputs a probability between the values of 0 and 1 that the image represents either a cheetah, a lion, or a tiger. The closer the value is to 1, the more confident the model is that its classification is correct.

One of the most common applications of image classification is facial recognition in personal photos.

AIXPRT uses the ResNet-50 network model for image classification tasks.

Object detection

Object detection is the process by which a trained convolutional neural network model analyzes a still or video image and identifies the presence and location of multiple objects. As with image-classification training, object detection training uses thousands of images to teach the model to recognize specific labels, or classes. Later, when a workload task presents the model with an inference request by asking for object detection, the model identifies the presence of one or more objects in the image, records the location of those objects, and outputs a probability between the values of 0 and 1 that each of the objects represents a learned class.

Low-latency, highly accurate object detection is especially critical for emerging technologies such as autonomous vehicles.

AIXPRT uses the SSD-MobileNet v1 network model for image classification tasks.

Recommender system

Recommender systems, or engines, are data filtering and analysis systems that suggest products or services based on what they learn from user preferences and behavior. AIXPRT uses a Wide and Deep network model for recommender system tasks. Wide and Deep networks combine the memorization efficiency of linear learning (wide) models with the generalized inference powers of neural network (deep) models.

Recommender systems are ubiquitous in modern life, and serve as the brains behind the suggestion and auto-complete features we see in streaming content services, social media, navigation, and targeted advertising.

On what types of systems can I run AIXPRT?

AIXPRT use cases cut across a wide range of hardware segments, including servers, desktops, laptops, and edge devices. Not all AIXPRT workloads and test configurations are applicable to each segment. In many cases, the ideal combination of test configuration variables remains an open question for ongoing research.

Supported operating systems

AIXPRT runs on Ubuntu 18.04 LTS or Windows 10 systems.

Minimum hardware requirements

The minimum CPU and GPU requirements vary by toolkit. Testers can find the package-specific system requirements and installation instructions for each framework in the readme files included in the AIXPRT install package. The readme files for each respective framework in the download packages are located here:

• AIXPRT_1.0.1_OpenVINO_Windows\AIXPRT\Modules\Deep-Learning
• AIXPRT_1.0_OpenVINO_Ubuntu\AIXPRT\Modules\Deep-Learning
• AIXPRT_1.0_Tensorflow_Windows\AIXPRT\Modules\Deep-Learning
• AIXPRT_1.0_Tensorflow_Ubuntu\AIXPRT\Modules\Deep-Learning
• AIXPRT_1.0_TensorRT_Windows\AIXPRT\Modules\Deep-Learning
• AIXPRT_1.0_TensorRT_Ubuntu\AIXPRT\Modules\Deep-Learning
• AIXPRT_1.0_MXNet_Ubuntu\AIXPRT\Modules\Deep-Learning

You can also access the readme files in the AIXPRT Resources GitHub repository. Please be sure to read the “Known Issues” section in the readme, as there may be issues relevant to your specific configuration.

NOTE: The OpenVINO on Windows package includes a precompiled version of OpenVINO for easy installation on Windows via a few quick commands, and a script that installs the necessary OpenVINO dependencies.

Skills you’ll need

The skills needed to install and run AIXPRT successfully vary depending on the installation package. All AIXPRT test packages require basic terminal skills. The TensorRT packages require familiarity with setting up environment variables, working within Visual Studio (in Windows), and building software from source.

How long it takes

The length of a test run depends on the system under test and the test configuration settings. Test length can range from just a few minutes to several hours.

Choosing an AIXPRT download package

AI workloads are now relevant to all types of hardware, from servers to laptops to IOT devices, so we intentionally designed AIXPRT to support a wide range of potential hardware, toolkit, and workload configurations. This approach provides AIXPRT testers with a tool that is flexible enough to adapt to a variety of environments. The downside is that the number of options makes it complicated to determine which AIXPRT download package suits your needs.

To help testers navigate this complexity, we’ve developed an interactive package selector tool. Testers select options in five categories: operating system, host hardware, toolkit, target hardware, and workload. They can proceed in any order but must make a selection for every category. Because not all combinations work together, each selection the tester makes eliminates some options in the remaining categories.

After a tester selects an option, a check mark appears on the category icon, and the selection they have made appears in the category box (e.g., TensorFlow in the Toolkit category). This shows testers which categories they’ve completed and the selections they’ve made. After a tester completes more than one category, a Start over button appears in the lower-left corner. Clicking this button clears all selections and gives testers a clean slate.

Once you’ve completed all five categories, a Download button appears in the lower-right corner. When you click this, a popup appears that provides a link for the correct download package and associated readme file.

Testers who know exactly which package they need can bypass the tool and go directly to the download table.

Adjusting test variables

AIXPRT testers can adjust the following test configuration variables:

• Precision (FP32, FP16, or INT8)
• Batch size (1, 2, 4, 8, 16, 32, etc.)
• Number of concurrent instances (1 or more depending on hardware support)

Using alternative test configuration files

AIXPRT testers can adjust batch size, level of precision, and number of concurrent instances by editing the JSON file in the AIXPRT/Config directory. While the process is straightforward, editing each of the variables in a config file can take some time, and testers don’t always know the appropriate values for their system. To address both issues, we are offering a selection of alternative config files that testers can download and drop into the AIXPRT/Config directory. To access the alternative config files, visit the AIXPRT public resources repository.

Targeting hardware components

Depending on the operating system, toolkit, and workload, AIXPRT can target x86 CPUs, AMD discrete GPUs, Intel processor graphics, Intel Neural Compute Sticks, or NVIDIA GPUs.

Latency

In the context of machine learning, latency refers to the amount of time it takes to process one inference request.

What is a good latency result?

In real-time or near real-time use cases such as performing image recognition on individual photos being captured by a camera, lower latency is important because it improves the user experience. In other cases, such as performing image recognition on a large library of photos, achieving higher throughput might be preferable; designating larger batch sizes or running concurrent instances might allow the overall workload to complete more quickly.

The dynamics of these performance tradeoffs mean that there is no single “good” score for all machine learning scenarios. Some testers might prefer lower latency, while others would sacrifice latency to achieve the higher level of throughput that their use case demands.

When latency is your top priority

Suppose you’re a data science firm helping a client optimize manufacturing yield by speeding the inspection process. Cameras photograph each product and upload pictures to a server with software that queries a machine learning model. Because your goal is to identify defective items as quickly as possible, your priority is latency, and you could adjust the AIXPRT variables to use small batch sizes and a single instance.

Throughput

In the context of machine learning, throughput refers to the number of inputs a system processes in a given time period.

What is a good throughput result?

In real-time or near real-time use cases such as performing image recognition on individual photos being captured by a camera, lower latency is important because it improves the user experience. In other cases, such as performing image recognition on a large library of photos, achieving higher throughput might be preferable; designating larger batch sizes or running concurrent instances might allow the overall workload to complete more quickly.

The dynamics of these performance tradeoffs mean that there is no single “good” score for all machine learning scenarios. Some testers might prefer lower latency, while others would sacrifice latency to achieve the higher level of throughput that their use case demands.

When throughput is your top priority

Suppose you’re an archivist with hundreds of thousands of historical photographs you must categorize. Because your goal is to process this large volume of images efficiently, you care most about throughput, and you could adjust the AIXPRT variables to use large batch sizes and run concurrent instances.

Why does this matter for servers and cloud instances?

Currently, servers and cloud instances handle the bulk of machine learning inference work for many common apps. For app publishers, the process of choosing an ideal back-end setup can be difficult. Depending on the purpose of an app, publishers may want to prioritize high latency, high throughput, or a finely tuned balance between the two. Regardless of the priority, AIXPRT provides users with accurate and reliable data about these critical inference measures. That data can help businesses make successful infrastructure decisions and avoid costly periods of trial and error or customer dissatisfaction.

Why does this matter for PCs?

As desktop and laptop computing power continues to grow, an increasing number of app publishers are moving inference workloads from the back end to the client side. With inference work on the client side, apps can provide users with features such as voice and facial recognition, image classification, object detection, recommendations, and pattern recognition, even while offline or in areas that have slow or spotty network connections. Users that rely on these powerful apps need to know if their gear can handle the load, and AIXPRT can help identify which PCs are up to the task.

At AIXPRT.com, interested parties can view test results published by the BenchmarkXPRT Development Community. The following tips can help visitors navigate the results viewer:

• You can click the tabs at the top of the table to switch from ResNet-50 network results to SSD-MobileNet or Wide and Deep network results.
• You can click the header of any column to sort the data on that variable. One click sorts A-Z and two clicks sort Z-A.
• You can click the link in the Source column to visit a detailed page on that result. The page contains additional test configuration and system hardware information and lets you download results files.
• You can filter results in categories such as framework, target hardware, batch size, and precision, and can designate minimum throughput and maximum latency scores. When you select a value from a drop-down menu or enter text, the results change immediately to reflect the filter.
• You can search for variables such as processor vendor or processor speed.
• The viewer displays eight results per page by default. You can change this to 16, 48, or Show all.

Publishing results yourself

Anyone who runs an AIXPRT test is free to publish their results themselves at any time. There is no approval process nor any restrictions for publishing results.

Submitting results for inclusion in the AIXPRT results database

We invite and encourage everyone to submit benchmark results from their testing for inclusion in the public results table. To do so, please follow the instructions on the AIXPRT results submission page.

Neural networks

In the context of machine learning, artificial neural networks (ANNs) are computing systems designed to loosely imitate the types of biological neural networks seen in the brains of animals and humans.

Deep neural networks (DNNs) use many layers of artificial neurons (nodes) to process mathematical algorithms designed to convert an input (e.g., an image set containing multiple unidentified animals) into a desired output (e.g., an image set containing attached labels that accurately identify each animal).

Convolutional neural networks (CNNs) are DNNs in which each node in one layer connects to every node in the next layer. Many applications use CNNs for complex image analysis tasks.

Training

In the context of machine learning, “training” is the process by which a neural network learns how to complete a task (i.e., image classification) through the use of a training dataset (e.g., a set of images).

Interference

In the context of machine learning, inference refers to the process by which already-trained networks apply their capabilities in the real world, such as recognizing a cat in a picture that the model has never seen before.

Latency

In the context of machine learning, latency refers to the amount of time it takes to process one inference request.

Throughput

In the context of machine learning, throughput refers to the number of inputs a system processes in a given time period.

Image classification

Image classification, or recognition, is the process by which a trained convolutional neural network model analyzes an image and predicts (infers) what the image represents. For example, during training, the model may analyze thousands of images depicting three types of big cats: cheetahs, lions, and tigers. The model learns to associate specific properties of those images with each of those three labels. Later, when a workload task presents the model with an inference request by asking for classification of a picture of a big cat, the model outputs a probability between the values of 0 and 1 that the image represents either a cheetah, a lion, or a tiger. The closer the value is to 1, the more confident the model is that its classification is correct.

One of the most common applications of image classification is facial recognition in personal photos.

AIXPRT uses the ResNet-50 network model for image classification tasks.

Object detection

Object detection is the process by which a trained convolutional neural network model analyzes a still or video image and identifies the presence and location of multiple objects. As with image-classification training, object detection training uses thousands of images to teach the model to recognize specific labels, or classes. Later, when a workload task presents the model with an inference request by asking for object detection, the model identifies the presence of one or more objects in the image, records the location of those objects, and outputs a probability between the values of 0 and 1 that each of the objects represents a learned class.

Low-latency, highly accurate object detection is especially critical for emerging technologies such as autonomous vehicles.

AIXPRT uses the SSD-MobileNet v1 network model for image classification tasks.

Recommender system

Recommender systems, or engines, are data filtering and analysis systems that suggest products or services based on what they learn from user preferences and behavior. AIXPRT uses a Wide and Deep network model for recommender system tasks. Wide and Deep networks combine the memorization efficiency of linear learning (wide) models with the generalized inference powers of neural network (deep) models.

Recommender systems are ubiquitous in modern life, and serve as the brains behind the suggestion and auto-complete features we see in streaming content services, social media, navigation, and targeted advertising.

The problems that ML workloads solve vary considerably, and the hardware components that each workload primarily stresses depends on the types of preprocessing involved, the structure of the model, and the how the applications distribute data. Most ML workloads are compute intensive, which means that the more powerful the CPUs in your server clusters, the better performance you’ll see.