PT-Logo
Forgot your password?
BenchmarkXPRT Blog banner

Category: Windows

HDXPRT 4 v1.2 and the HDXPRT 4 source code package are available

This week, we have good news for HDXPRT 4 testers. A few weeks ago, we discussed the fact that Adobe removed the trial version of Adobe Photoshop Elements (PSE) 2018 from the PSE download page. HDXPRT 4 used PSE 2018 for the Edit Photos scenario, so this change meant that new HDXPRT testers would not be able to successfully install and run the benchmark.

Fortunately, we were able to adapt the Edit Photos scripts to use the new trial version of PSE 2020, and have incorporated those changes in an updated HDXPRT 4 build (v1.2). It’s available for download on HDXPRT.com, along with an updated user manual. Apart from slightly different instructions for installing the trial version of PSE 2020, all aspects of the installation and test process remain the same. We tested the new build and found that individual workload and overall scores did not vary significantly, so scores from the new build will be comparable to existing HDXPRT 4 scores.

We also posted the HDXPRT 4 source code and build instructions on the HDXPRT tab in the Members’ Area (login required). If you’d like to review XPRT source code, but haven’t yet joined the community, we encourage you to join! Registration is quick and easy, and if you work for a company or organization with an interest in benchmarking, you can join for free. Simply fill out the form with your company e-mail address and select the option to be considered for a free membership. We’ll contact you to verify the address and then activate your membership.

We apologize to HDXPRT testers for the inconvenience over the last several weeks, and we thank you for your patience while we worked on a solution. If you have any questions about HDXPRT or the community, please feel free to ask!

Justin

Understanding AIXPRT’s default number of requests

A few weeks ago, we discussed how AIXPRT testers can adjust the key variables of batch size, levels of precision, and number of concurrent instances by editing the JSON test configuration file in the AIXPRT/Config directory. In addition to those key variables, there is another variable in the config file called “total_requests” that has a different default setting depending on the AIXPRT test package you choose. This setting can significantly affect a test run, so it’s important for testers to know how it works.

The total_requests variable specifies how many inference requests AIXPRT will send to a network (e.g., ResNet-50) during one test iteration at a given batch size (e.g., Batch 1, 2, 4, etc.). This simulates the inference demand that the end users place on the system. Because we designed AIXPRT to run on different types of hardware, it makes sense to set the default number of requests for each test package to suit the most likely hardware environment for that package.

For example, testing with OpenVINO on Windows aligns more closely with a consumer (i.e., desktop or laptop) scenario than testing with OpenVINO on Ubuntu, which is more typical of server/datacenter testing. Desktop testers require a much lower inference demand than server testers, so the default total_requests settings for the two packages reflect that. The default for the OpenVINO/Windows package is 500, while the default for the OpenVINO/Ubuntu package is 5,000.

Also, setting the number of requests so low that a system finishes each workload in less than 1 second can produce high run-to-run variation, so our default settings represent a lower boundary that will work well for common test scenarios.

Below, we provide the current default total_requests setting for each AIXPRT test package:

  • MXNet: 1,000
  • OpenVINO Ubuntu: 5,000
  • OpenVINO Windows: 500
  • TensorFlow Ubuntu: 100
  • TensorFlow Windows: 10
  • TensorRT Ubuntu: 5,000
  • TensorRT Windows: 500


Testers can adjust these variables in the config file according to their own needs. Finding the optimal combination of machine learning variables for each scenario is often a matter of trial and error, and the default settings represent what we think is a reasonable starting point for each test package.

To adjust the total_requests setting, start by locating and opening the JSON test configuration file in the AIXPRT/Config directory. Below, we show a section of the default config file (CPU_INT8.json) for the OpenVINO-Windows test package (AIXPRT_1.0_OpenVINO_Windows.zip). For each batch size, the total_requests setting appears at the bottom of the list of configurable variables. In this case, the default setting Is 500. Change the total_requests numerical value for each batch size in the config file, save your changes, and close the file.

Total requests snip

Note that if you are running multiple concurrent instances, OpenVINO and TensorRT automatically distribute the number of requests among the instances. MXNet and TensorFlow users must manually allocate the instances in the config file. You can find an example of how to structure manual allocation here. We hope to make this process automatic for all toolkits in a future update.

We hope this information helps you understand the total_requests setting, and why the default values differ from one test package to another. If you have any questions or comments about this or other aspects of AIXPRT, please let us know.

Justin

AIXPRT is here!

We’re happy to announce that AIXPRT is now available to the public! AIXPRT includes support for the Intel OpenVINO, TensorFlow, and NVIDIA TensorRT toolkits to run image-classification and object-detection workloads with the ResNet-50 and SSD-MobileNet v1networks, as well as a Wide and Deep recommender system workload with the Apache MXNet toolkit. The test reports FP32, FP16, and INT8 levels of precision.

To access AIXPRT, visit the AIXPRT download page. There, a download table displays the AIXPRT test packages. Locate the operating system and toolkit you wish to test and click the corresponding Download link. For detailed installation instructions and information on hardware and software requirements for each package, click the package’s Readme link. If you’re not sure which AIXPRT package to choose, the AIXPRT package selector tool will help to guide you through the selection process.

In addition, the Helpful Info box on AIXPRT.com contains links to a repository of AIXPRT resources, as well links to XPRT blog discussions about key AIXPRT test configuration settings such as batch size and precision.

We hope AIXPRT will prove to be a valuable tool for you, and we’re thankful for all the input we received during the preview period! If you have any questions about AIXPRT, please let us know.

A necessary update for HDXPRT 4

If you tried to install HDXPRT 4 over the past few days, you likely noticed that Adobe Photoshop Elements 2018, the version the Edit Photos scenario uses, is no longer available on the Adobe Photoshop Elements download page. In the past, Adobe has provided access to multiple older versions of their software for some time after a new release, but they appear to be moving away from that practice. We have not yet found an alternative way for users to download PSE 2018 on a trial basis. Unfortunately, this means testers will be temporarily unable to successfully complete the HDXPRT 4 installation process.

We’re adapting the scripts in the HDXPRT 4 Edit Photos scenario to use PSE 2020. As soon as we finish, we’ll start testing, with a focus on determining whether the change significantly affects the individual workload or overall scores.

We apologize for the inconvenience that this issue causes for HDXPRT testers. We’ll continue to update the community here in the blog about our progress with the new build. If you have any questions or comments, please let us know.

Justin

An update on AIXPRT development

It’s been a while since we last discussed the AIXPRT Community Preview 3 (CP3) release schedule, so we want to let everyone know where things stand. Testing for CP3 has taken longer than we predicted, but we believe we’re nearly ready for the release.

Testers can expect three significant changes in AIXPRT CP3. First, we updated support for the Ubuntu test packages. During the initial development phase of AIXPRT, Ubuntu version 16.04 LTS (Long Term Support) was the most current LTS version, but version 18.04 is now available.

Second, we have added TensorRT test packages for Windows and Ubuntu. Previously, AIXPRT testers could test only the TensorFlow variant of TensorRT. Now, they can use TensorRT to test systems with NVIDIA GPUs.

Third, we have added the Wide and Deep recommender system workload with the MXNet toolkit. Recommender systems are AI-based information-filtering tools that learn from end user input and behavior patterns and try to present them with optimized outputs that suit their needs and preferences. If you’ve used Netflix, YouTube, or Amazon accounts, you’ve encountered recommender systems that learn from your behavior.

Currently, the recommender system workload in AIXPRT CP3 is available for Ubuntu testing, but not for Windows. Recommender system inference workloads typically run on datacenter hardware, which tends to be Linux based. If enough community members are interested in running the MXNet/Wide and Deep test package on Windows, we can investigate what that would entail. If you’d like to see that option, please let us know.

As always, if you have any questions about the AIXPRT development process, feel free to ask!

Justin

An updated HDXPRT 4 v1.1 installer package

Today, we published an updated HDXPRT 4 v1.1 installer package that addresses an issue brought to light by HDXPRT testers and our own follow-up testing. We’ve also encountered an issue caused by anti-virus program interference during the HDXPRT installation process, so we’re providing steps for a workaround below. Neither the updated build nor the workaround steps affect the comparability of previous HDXPRT 4 test scores.

The first issue involves the hdxprt4.exe setup file. You may recall that the main updates in HDXPRT 4 v1.1 were the inclusion of the latest version of HandBrake and the ability for testers to choose whether to target a system’s discrete graphics card during the Convert Videos workload. Prior to today’s update, the HDXPRT 4 v1.1 installation package mistakenly included an old hdxprt4.exe setup file, which likely caused problems for testers attempting to target discrete graphics. We apologize for this oversight. The installer package we published today includes the correct hdxprt4.exe setup file.

The second issue is that during the installation process, Windows Security and other anti-virus programs may quarantine some of the AutoIt executables that HDXPRT 4 uses to install real-world applications, and the incomplete installation process will cause the test to fail. The files do not contain viruses, but the anti-virus programs may assume that the user has not granted HDXPRT permission to install the ancillary files. One of the executables currently triggering this behavior is the MediaEspresso ME75_2x4K_transcode.exe file. To check whether your test system is quarantining this file, navigate to the C:\Program Files (x86)\HDXPRT4\HDXPRT4_Workloads\HDXPRT4_Tests folder. Once the installation process is complete, the folder should contain 32 files, including ME75_2x4K_transcode.exe. If you see all 32 files, you’re ready to test. (Note: Once you run the test, HDXPRT 4 will add HDXPRTRunLog.txt to the folder, so you might see 33 files.)

If you see only 31 files, ME75_2x4K_transcode.exe is likely missing. To restore it, use the following steps:

1. Open the Windows Security app.
2. Select Virus & threat protection.
3. Under Current threats, select Protection history.
4. Check to see if Windows Security removed any threats around the time you installed HDXPRT 4.
5. If so, click the drop-down menu on the right side, where Windows Security lists the severity of the threat, and look for a false positive that reports the ME75_2x4K_transcode.exe file as Trojan:Win32/Wacatac.B!ml.
6. Click the Actions drop-down menu, and select Restore.
7. Navigate to the C:\Program Files (x86)\HDXPRT4\HDXPRT4_Workloads\HDXPRT4_Tests folder, and check to see where the ME75_2x4K_transcode.exe file is present.

Windows Security and other anti-virus programs may quarantine other HDXPRT installation files in the future. If your first HDXPRT 4 run fails to complete successfully, we suggest checking the anti-virus quarantine for HDXPRT-related files.

We also updated the HDXPRT 4 User Manual to include the steps above. If you have any questions about any of these topics, please feel free to contact us.

Justin

Check out the other XPRTs: