Category: AI

A first look at the upcoming AIXPRT learning tool

on October 29, 2020

Last month, we announced that we’re working on a new AIXPRT learning tool. Because we want tech journalists, OEM lab engineers, and everyone who is interested in AIXPRT to be able to find the answers they need in as little time as possible, we’re designing this tool to serve as an information hub for common AIXPRT topics and questions.

We’re still finalizing aspects of the tool’s content and design, so some details may change, but we can now share a sneak peak of the main landing page. In the screenshot below, you can see that the tool will feature four primary areas of content:

The FAQ section will provide quick answers to the questions we receive most from testers and the tech press.
The AIXPRT basics section will describe specific topics such as the benchmark’s toolkits, networks, workloads, and hardware and software requirements.
The testing and results section will cover the testing process, the metrics the benchmark produces, and how to publish results.
The AI/ML primer will provide brief, easy-to-understand definitions of key AI and ML terms and concepts for those who want to learn more about the subject.

We’re excited about the new AIXPRT learning tool, and will share more information here in the blog as we get closer to a release date. If you have any questions about the tool, please let us know!

Justin

Posted in AI, AIXPRT, benchmark, Benchmark metrics, BenchmarkXPRT, Cross-platform benchmarks, Interactive tools, Machine learning, OpenVINO, Performance benchmarking, results, TensorFlow, TensorRT |

Following up

By Justin Greene

on October 1, 2020

This week, we’re sharing news on two topics that we’ve discussed here in the blog over the past several months: CloudXPRT v1.01 and a potential AIXPRT OpenVINO update.

CloudXPRT v1.01

Last week, we announced that we were very close to releasing an updated CloudXPRT build (v1.01) with two minor bug fixes, an improved post-test results processing script, and an adjustment to one of our test configuration recommendations. Our testing and prep is complete, and the new version is live in the CloudXPRT GitHub repository and on our site!

None of the v1.01 changes affect performance or test results, so scores from the new build are comparable to those from previous CloudXPRT builds. If you’d like to know more about the changes, take a look at last week’s blog post.

The AIXPRT OpenVINO update

In late July, we discussed our plans to update the AIXPRT OpenVINO packages with OpenVINO 2020.3 Long-Term Support (LTS). While there are no known problems with the existing AIXPRT OpenVINO package, the LTS version targets environments that benefit from maximum stability and don’t require a constant stream of new tools and feature changes, so we thought it would be well suited for a benchmark like AIXPRT.

We initially believed that the update process would be relatively simple, and we’d be able to release a new AIXPRT OpenVINO package in September. However, we’ve discovered that the process is involved enough to require substantial low-level recoding. At this time, it’s difficult to estimate when the updated build will be ready for release. For any testers looking forward to the update, we apologize for the delay.

If you have any questions or comments about these or any other XPRT-related topics, please let us know!

Justin

Posted in AI, AIXPRT, benchmark, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Cloud, CloudXPRT, Datacenter, GitHub, Machine learning, OpenVINO, Performance benchmarking | Tagged AIXPRT, cloud, CloudXPRT, GitHub, Intel OpenVINO, OpenVINO |

We’re working on an AIXPRT learning tool

By Justin Greene

on September 10, 2020

For anyone interested in learning more about AIXPRT, the Introduction to AIXPRT white paper provides detailed information about its toolkits, workloads, system requirements, installation, test parameters, and results. However, for AIXPRT.com visitors who want to find the answers to specific AIXPRT-related questions quickly, a white paper can be daunting.

Because we want tech journalists, OEM lab engineers, and everyone who is interested in AIXPRT to be able to find the answers they need in as little time as possible, we’ve decided to develop a new learning tool that will serve as an information hub for common AIXPRT topics and questions.

The new learning tool will be available online through our site. It will offer quick bites of information about the fundamentals of AIXPRT, why the benchmark matters, the benefits of AIXPRT testing and results, machine learning concepts, key terms, and practical testing concerns.

We’re still working on the tool’s content and design. Because we’re designing this tool for you, we’d love to hear the topics and questions you think we should include. If you have any suggestions, please let us know!

Justin

Posted in AI, AIXPRT, benchmark, Benchmarking, Cross-platform benchmarks, Interactive tools, Machine learning, White papers | Tagged AI, AIXPRT, benchmark, deep learning, machine learning |

Potential web technology additions for WebXPRT 4

By Justin Greene

on August 20, 2020

A few months ago, we invited readers to send in their thoughts and ideas about web technologies and workload scenarios that may be a good fit for the next WebXPRT. We’d like to share a few of those ideas today, and we invite you to continue to send your feedback. We’re approaching the time when we need to begin firming up plans for a WebXPRT 4 development cycle in 2021, but there’s still plenty of time for you to help shape the future of the benchmark.

One of the most promising ideas for WebXPRT 4 is the potential addition of one or more WebAssembly (WASM) workloads. WASM is a low-level, binary instruction format that works across all modern browsers. It offers web developers a great deal of flexibility and provides the speed and efficiency necessary for running complex client applications in the browser. WASM enables a variety of workload scenario options, including gaming, video editing, VR, virtual machines, image recognition, and interactive educational content.

In addition, the Chrome team is dropping Portable Native Client (PNaCL) support in favor of WASM, which is why we had to remove a PNaCL workload when updating CrXPRT 2015 to CrXPRT 2. We generally model CrXPRT workloads on existing WebXPRT workloads, so familiarizing ourselves with WASM could ultimately benefit more than one XPRT benchmark.

We are also considering adding a web-based machine learning workload with TensorFlow for JavaScript (TensorFlow.js). TensorFlow.js offers pre-trained models for a wide variety of tasks including image classification, object detection, sentence encoding, natural language processing, and more. We could also use this technology to enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.

Other ideas include using a WebGL-based workload to target GPUs and investigating ways to incorporate a battery life test. What do you think? Let us know!

Justin

Posted in AI, battery life, benchmark, BenchmarkXPRT, BenchmarkXPRT development community, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, image classification, Machine learning, object detection, Performance benchmarking, Virtual reality, Web-based testing, WebAssembly, WebGL, WebXPRT, WebXPRT 3, What makes a good benchmark? | Tagged benchmark, BenchmarkXPRT, browser, Chrome, gaming, image classification, machine learning, object detection, Performance, PNaCl, WASM, WebAssembly, WebXPRT |

The ongoing evolution of the BenchmarkXPRT Development Community

By Justin Greene

on August 6, 2020

This November will mark the tenth anniversary of the BenchmarkXPRT Development Community, which we originally called the HDXPRT Development Community. Since the early days of HDXPRT, our community has grown to include about 275 members from over 85 companies and organizations, and we’ve added seven benchmarks to the XPRT family. We initially mailed HDXPRT DVDs to testers interested in a new way to evaluate PC performance, and now thousands of users around the world download our benchmarks and rely on them to help measure the performance of everything from tablets to laptops to high-end datacenter hardware.

As the XPRTs continue to grow and evolve, we’ve worked to make sure that the resources that we offer—and the ways we offer them—continue to meet the needs of XPRT testers and community members. As we expand in the AI and datacenter spaces with AIXPRT and CloudXPRT, our user group is becoming larger and more diverse than ever. We have already made some changes to better serve this expanding group, and will be making additional changes over the months ahead.

The first set of changes relate to our community membership model. Originally, membership in the BenchmarkXPRT Development Community required a $20 fee and provided access to preview versions of new benchmarks, the ability to submit ideas for future benchmarks, and regular updates through our monthly newsletter and community announcements. To remove the financial obstacle to joining, we introduced a fee waiver process a few years ago.

Also, we know that some OEM employees and members of the tech press are interested in the XPRTs, but are unable to join the community for one reason or another. With these people in mind, we recently experimented with making the CloudXPRT Preview publicly available. Releasing preview builds to all who are interested makes it more likely that users will incorporate the XPRTs into their test suites, and we have decided to adopt this practice for other benchmarks going forward.

In the coming months, we’ll be updating parts of our website to increase access to XPRT content. For example, certain content such as source code for most of the XPRTs is currently available only to members. We plan to remove the login requirement for access to this material.

Please keep in mind that membership in the BenchmarkXPRT Development Community continues to offer exclusive opportunities. Members can join groups such as the CloudXPRT Results Review Group and offer direct input into the design of future benchmarks. Members also receive our monthly newsletters.

If you have any questions about the XPRTs or community membership, please feel free to ask!

Justin

Posted in AI, AIXPRT, benchmark, BenchmarkXPRT, BenchmarkXPRT development community, Cloud, CloudXPRT, Community Preview, Datacenter, HDXPRT, Performance benchmarking, Source code | Tagged AI, AIXPRT, benchmark, BenchmarkXPRT, cloud, CloudXPRT, datacenter, HDXPRT |

We’re working on an update for the AIXPRT OpenVINO workload

By Justin Greene

on July 23, 2020

Shortly after the initial AIXPRT release, we noted that each of the toolkits AIXPRT uses (Intel OpenVINO, TensorFlow, NVIDIA TensorRT, and Apache MXNet) is on its own development schedule, and new versions will sometimes appear with little warning. When this happens, we’ll have to respond by updating specific AIXPRT installation packages, giving AIXPRT testers relatively short notice.

This is one of those times! Intel recently released OpenVINO 2020.3 Long-Term Support (LTS), and we’re planning to update the AIXPRT OpenVINO packages with the LTS version. The LTS version targets environments that benefit from maximum stability, and don’t require a constant stream of new tools and feature changes. In other words, it’s well suited for a benchmark, and we think it’s a good fit for AIXPRT moving forward.

We don’t yet know what impact the new version will have on AIXPRT OpenVINO test results. A substantial part of the development process will involve testing the new packages on a variety of platforms to see how performance changes. We’ll communicate our findings here in the blog, so AIXPRT testers will know what to expect.

Thankfully, the modular nature of the AIXPRT installation packages ensures that we don’t need to revise the entire AIXPRT suite every time a toolkit update goes live. If you test with only TensorFlow, TensorRT, or MXNet, or a combination of those toolkits, this update won’t affect your testing.

We’re not ready to commit to a release date for the new build, but anticipate it will be in September.

If you have any questions about AIXPRT or OpenVINO, please let us know!

Justin

Posted in AI, AIXPRT, benchmark, BenchmarkXPRT, BenchmarkXPRT development community, MXNet, OpenVINO, TensorFlow, TensorRT | Tagged AIXPRT, Apache MXNet, Intel OpenVINO, NVIDIA TensorRT, OpenVINO, TensorRT |

Category: AI

A first look at the upcoming AIXPRT learning tool

Following up

CloudXPRT v1.01

The AIXPRT OpenVINO update

We’re working on an AIXPRT learning tool

Potential web technology additions for WebXPRT 4

The ongoing evolution of the BenchmarkXPRT Development Community

We’re working on an update for the AIXPRT OpenVINO workload

Check out the other XPRTs: