BenchmarkXPRT Blog banner

Tag Archives: Windows

Testing XPRT compatibility with Windows 11

Last week, Microsoft announced that the Windows 11 GA build will officially launch Tuesday October 5, earlier than the initial late 2021 estimate. The update will start rolling out with select new laptops and existing Windows 10 PCs that satisfy specific system requirements, and only some Windows 10 PCs will be eligible for the update right away. Through a phased Windows Update process, additional Windows 10 PCs will be able to access the update throughout the first half of 2022.

Between the phased Windows 11 rollout and the pledge Microsoft has made to continue Windows 10 support through October 2025, it will likely be a while before the majority of Windows users transition to the new version. We hope the transition period will go smoothly for the XPRTs. However, because we designed three of our benchmarks to run on Windows 10 (HDXPRT 4, TouchXPRT 2016, and AIXPRT), we might encounter compatibility issues with Windows 11.

Over the coming weeks, we’ll be testing HDXPRT 4, TouchXPRT 2016, and AIXPRT on beta versions of Windows 11, and we’ll test again after the GA launch. In addition to obvious compatibility issues and test failures, we’ll note any changes we need to make to our documentation to account for differences in the Windows 11 installation or test processes.

We hope that testers will be able to successfully use all three benchmarks on both OS versions throughout the transition process. If problems arise, we will keep our blog readers informed while exploring solutions. As always, we’re also open to feedback from the community, so if you are participating in the Windows Insider Program and have encountered Windows 11 beta compatibility issues with any of the Windows-focused XPRTs, please let us know!

Justin

Understanding AIXPRT’s default number of requests

A few weeks ago, we discussed how AIXPRT testers can adjust the key variables of batch size, levels of precision, and number of concurrent instances by editing the JSON test configuration file in the AIXPRT/Config directory. In addition to those key variables, there is another variable in the config file called “total_requests” that has a different default setting depending on the AIXPRT test package you choose. This setting can significantly affect a test run, so it’s important for testers to know how it works.

The total_requests variable specifies how many inference requests AIXPRT will send to a network (e.g., ResNet-50) during one test iteration at a given batch size (e.g., Batch 1, 2, 4, etc.). This simulates the inference demand that the end users place on the system. Because we designed AIXPRT to run on different types of hardware, it makes sense to set the default number of requests for each test package to suit the most likely hardware environment for that package.

For example, testing with OpenVINO on Windows aligns more closely with a consumer (i.e., desktop or laptop) scenario than testing with OpenVINO on Ubuntu, which is more typical of server/datacenter testing. Desktop testers require a much lower inference demand than server testers, so the default total_requests settings for the two packages reflect that. The default for the OpenVINO/Windows package is 500, while the default for the OpenVINO/Ubuntu package is 5,000.

Also, setting the number of requests so low that a system finishes each workload in less than 1 second can produce high run-to-run variation, so our default settings represent a lower boundary that will work well for common test scenarios.

Below, we provide the current default total_requests setting for each AIXPRT test package:

  • MXNet: 1,000
  • OpenVINO Ubuntu: 5,000
  • OpenVINO Windows: 500
  • TensorFlow Ubuntu: 100
  • TensorFlow Windows: 10
  • TensorRT Ubuntu: 5,000
  • TensorRT Windows: 500


Testers can adjust these variables in the config file according to their own needs. Finding the optimal combination of machine learning variables for each scenario is often a matter of trial and error, and the default settings represent what we think is a reasonable starting point for each test package.

To adjust the total_requests setting, start by locating and opening the JSON test configuration file in the AIXPRT/Config directory. Below, we show a section of the default config file (CPU_INT8.json) for the OpenVINO-Windows test package (AIXPRT_1.0_OpenVINO_Windows.zip). For each batch size, the total_requests setting appears at the bottom of the list of configurable variables. In this case, the default setting Is 500. Change the total_requests numerical value for each batch size in the config file, save your changes, and close the file.

Total requests snip

Note that if you are running multiple concurrent instances, OpenVINO and TensorRT automatically distribute the number of requests among the instances. MXNet and TensorFlow users must manually allocate the instances in the config file. You can find an example of how to structure manual allocation here. We hope to make this process automatic for all toolkits in a future update.

We hope this information helps you understand the total_requests setting, and why the default values differ from one test package to another. If you have any questions or comments about this or other aspects of AIXPRT, please let us know.

Justin

A necessary update for HDXPRT 4

If you tried to install HDXPRT 4 over the past few days, you likely noticed that Adobe Photoshop Elements 2018, the version the Edit Photos scenario uses, is no longer available on the Adobe Photoshop Elements download page. In the past, Adobe has provided access to multiple older versions of their software for some time after a new release, but they appear to be moving away from that practice. We have not yet found an alternative way for users to download PSE 2018 on a trial basis. Unfortunately, this means testers will be temporarily unable to successfully complete the HDXPRT 4 installation process.

We’re adapting the scripts in the HDXPRT 4 Edit Photos scenario to use PSE 2020. As soon as we finish, we’ll start testing, with a focus on determining whether the change significantly affects the individual workload or overall scores.

We apologize for the inconvenience that this issue causes for HDXPRT testers. We’ll continue to update the community here in the blog about our progress with the new build. If you have any questions or comments, please let us know.

Justin

How to use alternate configuration files with AIXPRT

In last week’s AIXPRT Community Preview 3 announcement, we mentioned the new public GitHub repository that we’re using to publish AIXPRT-related information and resources. In addition to the installation readmes for each AIXPRT installation package, the repository contains a selection of alternative test config files that testers can use to quickly and easily change a test’s parameters.

As we discussed in previous blog entries about batch size, levels of precision, and number of concurrent instances, AIXPRT testers can adjust each of these key variables by editing the JSON file in the AIXPRT/Config directory. While the process is straightforward, editing each of the variables in a config file can take some time, and testers don’t always know the appropriate values for their system. To address both of these issues, we are offering a selection of alternative config files that testers can download and drop into the AIXPRT/Config directory.

In the GitHub repository, we’ve organized the available config files first by operating system (Linux_Ubuntu and Windows) and then by vendor (All, Intel, and NVIDIA). Within each section, testers will find preconfigured JSON files set up for several scenarios, such as running with multiple concurrent instances on a system’s CPU or GPU, running with FP32 precision instead of FP16, etc. The picture below shows the preconfigured files that are currently available for systems running Ubuntu on Intel hardware.

AIXPRT public repository snip 2

Because potential AIXPRT use cases cut across a wide range of hardware segments, including desktops, edge devices, and servers, not all AIXPRT workloads and configs will be applicable to each segment. As we move towards the AIXPRT GA, we’re working to find the best way to parse out these distinctions and communicate them to end users. In many cases, the ideal combination of test configuration variables remains an open question for ongoing research. However, we hope the alternative configuration files will help by giving testers a starting place.

If you experiment with an alternative test configuration file, please note that it should replace the existing default config file. If more than one config file is present, AIXPRT will run all the configurations and generate a separate result for each. More information about the config files and detailed instructions for how to handle the files are available in the EditConfig.md document in the public repository.

We’ll continue to keep everyone up to date with AIXPRT news here in the blog. If you have any questions or comments, please let us know.

Justin

AIXPRT Community Preview 3 is here!

We’re happy to announce that the AIXPRT Community Preview 3 (CP3) is now available! As we discussed in last week’s blog, testers can expect three significant changes in AIXPRT CP3:

  • We updated support for the Ubuntu test packages from Ubuntu version 16.04 LTS to version 18.04 LTS.
  • We added TensorRT test packages for Windows and Ubuntu. Previously, AIXPRT testers could test only the TensorFlow variant of TensorRT. Now, they can use TensorRT to test systems with NVIDIA GPUs.
  • We added the Wide and Deep recommender system workload with the MXNet toolkit for Ubuntu systems.


To access AIXPRT CP3, click this access link and submit the brief information form unless you’ve already done so for CP2. You will then gain access to the AIXPRT community preview page. (If you’re not already a BenchmarkXPRT Development Community member, we’ll contact you with more information about your membership.)

On the community preview page, a download table displays the currently available AIXPRT CP3 test packages. Locate the operating system and toolkit you wish to test, and click the corresponding Download link. For detailed installation instructions and information on hardware and software requirements for each package, click the corresponding Readme link. Instead of providing installation guide PDFs as we did for CP2, we are now directing testers to a public GitHub repository. The repository contains the installation readmes for all the test packages, as well as a selection of alternative test configuration files. We’ll discuss the alternative configuration files in more detail in a future blog post.

Note: Those who have access to the existing AIXPRT GitHub repository will be able to access CP3 in the same way as previous versions.

We’ll continue to keep everyone up to date with AIXPRT news here in the blog. If you have any questions or comments, please let us know.

Justin

An updated HDXPRT 4 v1.1 installer package

Today, we published an updated HDXPRT 4 v1.1 installer package that addresses an issue brought to light by HDXPRT testers and our own follow-up testing. We’ve also encountered an issue caused by anti-virus program interference during the HDXPRT installation process, so we’re providing steps for a workaround below. Neither the updated build nor the workaround steps affect the comparability of previous HDXPRT 4 test scores.

The first issue involves the hdxprt4.exe setup file. You may recall that the main updates in HDXPRT 4 v1.1 were the inclusion of the latest version of HandBrake and the ability for testers to choose whether to target a system’s discrete graphics card during the Convert Videos workload. Prior to today’s update, the HDXPRT 4 v1.1 installation package mistakenly included an old hdxprt4.exe setup file, which likely caused problems for testers attempting to target discrete graphics. We apologize for this oversight. The installer package we published today includes the correct hdxprt4.exe setup file.

The second issue is that during the installation process, Windows Security and other anti-virus programs may quarantine some of the AutoIt executables that HDXPRT 4 uses to install real-world applications, and the incomplete installation process will cause the test to fail. The files do not contain viruses, but the anti-virus programs may assume that the user has not granted HDXPRT permission to install the ancillary files. One of the executables currently triggering this behavior is the MediaEspresso ME75_2x4K_transcode.exe file. To check whether your test system is quarantining this file, navigate to the C:\Program Files (x86)\HDXPRT4\HDXPRT4_Workloads\HDXPRT4_Tests folder. Once the installation process is complete, the folder should contain 32 files, including ME75_2x4K_transcode.exe. If you see all 32 files, you’re ready to test. (Note: Once you run the test, HDXPRT 4 will add HDXPRTRunLog.txt to the folder, so you might see 33 files.)

If you see only 31 files, ME75_2x4K_transcode.exe is likely missing. To restore it, use the following steps:

1. Open the Windows Security app.
2. Select Virus & threat protection.
3. Under Current threats, select Protection history.
4. Check to see if Windows Security removed any threats around the time you installed HDXPRT 4.
5. If so, click the drop-down menu on the right side, where Windows Security lists the severity of the threat, and look for a false positive that reports the ME75_2x4K_transcode.exe file as Trojan:Win32/Wacatac.B!ml.
6. Click the Actions drop-down menu, and select Restore.
7. Navigate to the C:\Program Files (x86)\HDXPRT4\HDXPRT4_Workloads\HDXPRT4_Tests folder, and check to see where the ME75_2x4K_transcode.exe file is present.

Windows Security and other anti-virus programs may quarantine other HDXPRT installation files in the future. If your first HDXPRT 4 run fails to complete successfully, we suggest checking the anti-virus quarantine for HDXPRT-related files.

We also updated the HDXPRT 4 User Manual to include the steps above. If you have any questions about any of these topics, please feel free to contact us.

Justin

Check out the other XPRTs:

Forgot your password?