BenchmarkXPRT Blog banner

Category: Windows 10

Understanding AIXPRT results

Last week, we discussed the changes we made to the AIXPRT Community Preview 2 (CP2) download page as part of our ongoing effort to make AIXPRT easier to use. This week, we want to discuss the basics of understanding AIXPRT results by talking about the numbers that really matter and how to access and read the actual results files.

To understand AIXPRT results at a high level, it’s important to revisit the core purpose of the benchmark. AIXPRT’s bundled toolkits measure inference latency (the speed of image processing) and throughput (the number of images processed in a given time period) for image recognition (ResNet-50) and object detection (SSD-MobileNet v1) tasks. Testers have the option of adjusting variables such as batch size (the number of input samples to process simultaneously) to try and achieve higher levels of throughput, but higher throughput can come at the expense of increased latency per task. In real-time or near real-time use cases such as performing image recognition on individual photos being captured by a camera, lower latency is important because it improves the user experience. In other cases, such as performing image recognition on a large library of photos, achieving higher throughput might be preferable; designating larger batch sizes or running concurrent instances might allow the overall workload to complete more quickly.

The dynamics of these performance tradeoffs ensure that there is no single good score for all machine learning scenarios. Some testers might prefer lower latency, while others would sacrifice latency to achieve the higher level of throughput that their use case demands.

Testers can find latency and throughput numbers for each completed run in a JSON results file in the AIXPRT/Results folder. The test also generates CSV results files that are in the same folder. The raw results files report values for each AI task configuration (e.g., ResNet-50, Batch1, on CPU). Parsing and consolidating the raw data can take some time, so we’re developing a results file parsing tool to make the job much easier.

The results parsing tool is currently available in the AIXPRT CP2 OpenVINO – Windows package, and we hope to make it available for more packages soon. Using the tool is as simple as running a single command, and detailed instructions for how to do so are in the AIXPRT OpenVINO on Windows user guide. The tool produces a summary (example below) that makes it easier to quickly identify relevant comparison points such as maximum throughput and minimum latency.

AIXPRT results summary

In addition to the summary, the tool displays the throughput and latency results for each AI task configuration tested by the benchmark. AIXPRT runs each AI task multiple times and reports the average inference throughput and corresponding latency percentiles.

AIXPRT results details

We hope that this information helps to make it easier to understand AIXPRT results. If you have any questions or comments, please feel free to contact us.

Justin

Navigating the AIXPRT Community Preview download page just got easier

AIXPRT Community Preview 2 (CP2) has been generating quite a bit of interest among the BenchmarkXPRT Development Community and members of the tech press. We’re excited that the tool has piqued curiosity and that folks are recognizing its value for technical analysis. When talking with folks about test setup and configuration, we keep hearing the same questions:

  • How do I find the exact toolkit or package that I need?
  • How do I find the instructions for a specific toolkit?
  • What test configuration variables are most important for producing consistent, relevant results?
  • How do I know which values to choose when configuring options such as iterations, concurrent instances, and batch size?


In the coming weeks, we’ll be working to provide detailed answers to questions about test configuration. In response to the confusion about finding specific packages and instructions, we’ve redesigned the CP2 download page to make it easier for you to find what you need. Below, we show a snapshot from the new CP2 download table. Instead of having to download the entire CP2 package that includes the OpenVINO, TensorFlow, and TensorRT in TensorFlow test packages, you can now download one package at a time. In the Documentation column, we’ve posted package-specific instructions, so you won’t have to wade through the entire installation guide to find the instructions you need.

AIXPRT Community Preview download table

We hope these changes make it easier for people to experiment with AIXPRT. As always, please feel free to contact us with any questions or comments you may have.

Justin

A new HDXPRT 4 build is available!

A few weeks ago, we announced that a new HDXPRT 4 build, v1.1, was on the way. This past Monday, we published the build on HDXPRT.com.

The new build includes an updated version of HandBrake, the commercial application that HDXPRT uses for certain video conversion tasks. HandBrake 1.2.2 supports hardware acceleration with AMD Video Coding Engine (VCE), Intel Quick Sync, and the NVIDIA video encoder (NVENC). By default, HDXPRT4 v1.1 uses the encoder available through a system’s integrated graphics, but testers can target discrete graphics by changing a configuration file flag before running the benchmark. HDXPRT will then use the encoder provided by the discrete graphics hardware. This configuration setting takes effect only when more than one of the supported encoders (VCE, QSV, or NVENC) is present on the system.

As we mentioned before, in all other respects, the benchmark has not changed. That means that, apart from a scenario where a tester changes the targeted graphics hardware, scores from previous HDXPRT 4 builds will be comparable to those from the new build.

The updated HDXPRT 4 User Manual contains additional information and instructions for changing the configuration file flag. Please contact us if you have any questions about the new build. Happy testing!

Justin

Making AIXPRT easier to use

We’re glad to see so much interest in the AIXPRT CP2 build. Over the past few days, we’ve received two questions about the setup process: 1) where to find instructions for setting up AIXPRT on Windows, and 2) whether we could make it easier to install Intel OpenVINO on test systems.

In response to the first question, testers can find the relevant instructions for each framework in the readme files included in the AIXPRT install package. Instructions for Windows installation are in section 3 of the OpenVINO and TensorFlow readmes. Please note that whether you’re running AIXPRT on Ubuntu or Windows, be sure to read the “Known Issues” section in the readme, as there may be issues relevant to your specific configuration.

The readme files for each respective framework in the CP2 package are located here:

  • AIXPRT_0.5_CP2\AIXPRT_OpenVINO_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning
  • AIXPRT_0.5_CP2\AIXPRT_TensorFLow_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning
  • AIXPRT_0.5_CP2\AIXPRT_TensorFlow_TensorRT_0.5_CP2.zip\AIXPRT\Modules\Deep-Learning


We’re also working on consolidating the instructions into a central document that will make it easier for everyone to find the instructions they need.

In response to the question about OpenVINO installation, we’re working on an AIXPRT CP2 package that includes a precompiled version of OpenVINO R5.0.1 for easy installation on Windows via a few quick commands, and a script that installs the necessary OpenVINO dependencies. We’re currently testing the build, and we’ll make it available to testers as soon as possible.

The tests themselves will not change, so the new build will not influence existing results from Ubuntu or Windows. We hope it will simply facilitate the setup and testing process for many users.

We appreciate each bit of feedback that we receive, so if you have any suggestions for AIXPRT, please let us know!

Justin

We want to hear your thoughts about the AIXPRT development schedule

We released the second AIXPRT Community Preview (CP2) about two weeks ago. The main additions in CP2 were the ability to run certain test configurations in Windows (OpenVINO CPU/GPU and TensorFlow CPU), the option to download the installer package from the AIXPRT tab in the XPRT Members’ Area, and a demo mode.

We’re also investigating ways to support TensorFlow GPU and TensorFlow-TensorRT testing in Windows, and we’d like to eventually add support for TensorRT testing in Ubuntu and Windows. If development and pre-release testing go as planned, we may roll out some of these extra features by the end of June. However, it’s possible that getting all the pieces that we want in place will require a multi-step release process. If so, we’re considering two approaches: (1) issuing a third community preview (CP3) and (2) preparing a general availability (GA) release, to which we would add features over the months following the release. Neither of these paths is likely to affect test results from the currently supported configurations.

Would you like to work with another community preview, or would it be better for us to move straight to a GA release and add features as they become ready? We want to follow the approach that the majority of community members prefer, so please let us know what you think. As always, we also welcome any questions, concerns, or suggestions regarding the AIXPRT development process.

Justin

TouchXPRT: a great tool for evaluating Windows performance

From time to time, we remember that some XPRT users have experience with only one or two of the benchmark tools in our portfolio. They might have bookmarked a link to WebXPRT they found in a tech review or copied the HDXPRT installer package from a flash drive in their lab, but are unaware of other members of the XPRT family that could be useful to them. To spread the word on the range of capabilities the XPRTs offer, we occasionally highlight one of the XPRT tools in the blog . Last week, we discussed CrXPRT, a benchmark for evaluating the performance and battery life of Chrome OS devices. Today, we focus on TouchXPRT, our app for evaluating the performance of Windows 10 devices.

While our first benchmark, HDXPRT, is a great tool for assessing how well Windows machines handle media creation tasks using real commercial applications, it’s simply too large to run on most Windows tablets, 2-in-1s, and laptops with limited memory. To test those devices, we developed the latest version of TouchXPRT as a Universal Windows Platform app. As a Windows app, installing TouchXPRT is easy and quick (about 15 minutes). It runs five tests that simulate common photo, video, and music editing tasks; measures how quickly the device completes each of those tasks; and provides an overall score. It takes about 15 minutes to run on most devices. Labs can also automate testing using the command line or a script.

Want to run TouchXPRT?

Download TouchXPRT from the Microsoft Store or from TouchXPRT.com. The TouchXPRT 2016 release notes provide step-by-step instructions. To compare device scores, go to the TouchXPRT 2016 results page, where you’ll find scores from many Windows 10 devices.

Want to dig into the details?

Check out the Exploring TouchXPRT 2016 white paper. In it, we discuss the TouchXPRT development process, its component tests and workloads, and how it calculates individual workload and overall scores. We also provide instructions for automated testing.

BenchmarkXPRT Development Community members also have access to the TouchXPRT source code, so consider joining the community today. There’s no obligation and membership is free for members of any company or organization with an interest in benchmarks.

If you’ve been looking for a Windows performance evaluation tool that’s easy to use and has the flexibility of a UWP app, give TouchXPRT a try and let us know what you think!

Justin

Check out the other XPRTs:

Forgot your password?