BenchmarkXPRT Blog banner

Category: BenchmarkXPRT development community

Following up

This week, we’re sharing news on two topics that we’ve discussed here in the blog over the past several months: CloudXPRT v1.01 and a potential AIXPRT OpenVINO update.

CloudXPRT v1.01

Last week, we announced that we were very close to releasing an updated CloudXPRT build (v1.01) with two minor bug fixes, an improved post-test results processing script, and an adjustment to one of our test configuration recommendations. Our testing and prep is complete, and the new version is live in the CloudXPRT GitHub repository and on our site!

None of the v1.01 changes affect performance or test results, so scores from the new build are comparable to those from previous CloudXPRT builds. If you’d like to know more about the changes, take a look at last week’s blog post.

The AIXPRT OpenVINO update

In late July, we discussed our plans to update the AIXPRT OpenVINO packages with OpenVINO 2020.3 Long-Term Support (LTS). While there are no known problems with the existing AIXPRT OpenVINO package, the LTS version targets environments that benefit from maximum stability and don’t require a constant stream of new tools and feature changes, so we thought it would be well suited for a benchmark like AIXPRT.

We initially believed that the update process would be relatively simple, and we’d be able to release a new AIXPRT OpenVINO package in September. However, we’ve discovered that the process is involved enough to require substantial low-level recoding. At this time, it’s difficult to estimate when the updated build will be ready for release. For any testers looking forward to the update, we apologize for the delay.

If you have any questions or comments about these or any other XPRT-related topics, please let us know!

Justin

WebXPRT 3: relevant, reliable, and easy to use

WebXPRT continues to be the most widely-used XPRT benchmark, with just over 625,000 runs to date. From the first WebXPRT release in 2013, WebXPRT has been popular with device manufacturers, developers, tech journalists, and consumers because it’s easy to run, it runs on almost anything with a web browser, and its workloads reflect the types of web-based tasks that people are likely to encounter on a daily basis.

We realize that many folks who follow the XPRTs may be unaware of the wide variety of WebXPRT uses that we frequently read about in the tech press. Today, we thought it would be interesting to bring the numbers to life. In addition to dozens of device reviews, here’s a sample of WebXPRT 3 mentions over the past few weeks.

As we plan for the next version of WebXPRT, we want to be sure we build a benchmark that continues WebXPRT’s legacy of relevant workloads, ease-of-use, and broad compatibility. We know what works well in our lab, but to build a benchmark that meets the needs of a diverse group of users all around the world, it’s important that we hear from all types of testers. We recently discussed some of the new technologies that we’re considering for WebXPRT 4, so please don’t hesitate to let us know what you think about those proposals, or send any additional ideas you may have!

Justin

Coming soon: a white paper about the CloudXPRT web microservices workload

Soon, we’ll be expanding our portfolio of CloudXPRT resources with a white paper that focuses on the benchmark’s web microservices workload. While we summarized the workload in the Introduction to CloudXPRT white paper, the new paper will discuss the workload in much greater detail.

In addition to providing practical information about the web microservices installation packages and minimum system requirements, the paper describes the workload’s test configuration variables, structural components, task workflows, and test metrics. It also discusses interpreting test results and the process for submitting results for publication.

As we’ve noted, CloudXPRT is one of the more complex tools in the XPRT family, with no shortage of topics to explore further. We plan to publish a companion overview for the data analytics workload, and possible future topics include the impact of adjusting specific test configuration options, recommendations for results reporting, and methods for analysis.

We hope that the upcoming Overview of the CloudXPRT Web Microservices Workload paper will serve as a go-to resource for CloudXPRT testers, and will answer any questions you have about the workload. Once it goes live, we’ll provide links in the Helpful Info box on CloudXPRT.com and the CloudXPRT section of our XPRT white papers page.

If you have any questions, please let us know!

Justin

CloudXPRT version 1.0 is here!

The CloudXPRT Preview period has ended, and CloudXPRT version 1.0 installation packages are now available on CloudXPRT.com and the BenchmarkXPRT GitHub repository! Like the Preview build, CloudXPRT version 1.0 includes two workloads: web microservices and data analytics (you can find more details about the workloads here). Testers can use metrics from the workloads to compare IaaS stack (both hardware and software) performance and to evaluate whether any given stack is capable of meeting SLA thresholds. You can configure CloudXPRT to run on local datacenter, Amazon Web Services, Google Cloud Platform, or Microsoft Azure deployments.

Several different test packages are available for download from the CloudXPRT download page. For detailed installation instructions and hardware and software requirements for each, click the package’s readme link. On CloudXPRT.com, the Helpful Info box contains resources such as links to the Introduction to CloudXPRT white paper, the CloudXPRT master readme, and the CloudXPRT GitHub repository.

The GitHub repository also contains the CloudXPRT source code. The source code is freely available for testers to download and review.

Performance results from this release are comparable to performance results from the CloudXPRT Preview build. Testers who wish to publish results on CloudXPRT.com can find more information about the results submission and review process in the blog. We post the monthly results cycle schedule on the results submission page.

We’re thankful for all the input we received during the CloudXPRT development process and Preview period. If you have any questions about CloudXPRT, please let us know.

Justin

Potential web technology additions for WebXPRT 4

A few months ago, we invited readers to send in their thoughts and ideas about web technologies and workload scenarios that may be a good fit for the next WebXPRT. We’d like to share a few of those ideas today, and we invite you to continue to send your feedback. We’re approaching the time when we need to begin firming up plans for a WebXPRT 4 development cycle in 2021, but there’s still plenty of time for you to help shape the future of the benchmark.

One of the most promising ideas for WebXPRT 4 is the potential addition of one or more WebAssembly (WASM) workloads. WASM is a low-level, binary instruction format that works across all modern browsers. It offers web developers a great deal of flexibility and provides the speed and efficiency necessary for running complex client applications in the browser. WASM enables a variety of workload scenario options, including gaming, video editing, VR, virtual machines, image recognition, and interactive educational content.

In addition, the Chrome team is dropping Portable Native Client (PNaCL) support in favor of WASM, which is why we had to remove a PNaCL workload when updating CrXPRT 2015 to CrXPRT 2. We generally model CrXPRT workloads on existing WebXPRT workloads, so familiarizing ourselves with WASM could ultimately benefit more than one XPRT benchmark.

We are also considering adding a web-based machine learning workload with TensorFlow for JavaScript (TensorFlow.js). TensorFlow.js offers pre-trained models for a wide variety of tasks including image classification, object detection, sentence encoding, natural language processing, and more. We could also use this technology to enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.

Other ideas include using a WebGL-based workload to target GPUs and investigating ways to incorporate a battery life test. What do you think? Let us know!

Justin

The ongoing evolution of the BenchmarkXPRT Development Community

This November will mark the tenth anniversary of the BenchmarkXPRT Development Community, which we originally called the HDXPRT Development Community. Since the early days of HDXPRT, our community has grown to include about 275 members from over 85 companies and organizations, and we’ve added seven benchmarks to the XPRT family. We initially mailed HDXPRT DVDs to testers interested in a new way to evaluate PC performance, and now thousands of users around the world download our benchmarks and rely on them to help measure the performance of everything from tablets to laptops to high-end datacenter hardware.

As the XPRTs continue to grow and evolve, we’ve worked to make sure that the resources that we offer—and the ways we offer them—continue to meet the needs of XPRT testers and community members. As we expand in the AI and datacenter spaces with AIXPRT and CloudXPRT, our user group is becoming larger and more diverse than ever. We have already made some changes to better serve this expanding group, and will be making additional changes over the months ahead.

The first set of changes relate to our community membership model. Originally, membership in the BenchmarkXPRT Development Community required a $20 fee and provided access to preview versions of new benchmarks, the ability to submit ideas for future benchmarks, and regular updates through our monthly newsletter and community announcements. To remove the financial obstacle to joining, we introduced a fee waiver process a few years ago.

Also, we know that some OEM employees and members of the tech press are interested in the XPRTs, but are unable to join the community for one reason or another. With these people in mind, we recently experimented with making the CloudXPRT Preview publicly available. Releasing preview builds to all who are interested makes it more likely that users will incorporate the XPRTs into their test suites, and we have decided to adopt this practice for other benchmarks going forward.

In the coming months, we’ll be updating parts of our website to increase access to XPRT content. For example, certain content such as source code for most of the XPRTs is currently available only to members. We plan to remove the login requirement for access to this material.

Please keep in mind that membership in the BenchmarkXPRT Development Community continues to offer exclusive opportunities. Members can join groups such as the CloudXPRT Results Review Group and offer direct input into the design of future benchmarks. Members also receive our monthly newsletters.

If you have any questions about the XPRTs or community membership, please feel free to ask!

Justin

Check out the other XPRTs:

Forgot your password?