BenchmarkXPRT Blog banner

Category: BenchmarkXPRT development community

Looking back on 2021 with the XPRTs

Justin Greene
By Justin Greene
on January 6, 2022

As 2022 gets underway, we want to take this opportunity to look back on 2021 and review another productive year for the XPRTs. Readers of our newsletter are familiar with the stats and updates we include each month, but for our blog readers who don’t receive the newsletter, we’ve compiled some highlights below.

Benchmarks
In the past year, we released the WebXPRT 4 Preview, CloudXPRT v1.1, and an updated CrXPRT 2 build that included a fix for prior issues with the battery life test.

XPRTs in the media
Journalists, advertisers, and analysts referenced the XPRTs thousands of times in 2021. It’s always rewarding to know that the XPRTs have proven to be useful and reliable assessment tools for technology publications such as AnandTech, Expert Reviews, Gadgets 360, Gizmodo, Hot Hardware, Laptop Mag, Legit Reviews, Notebookcheck, PCMag, PCWorld, TechPowerUp, Tom’s Hardware, and ZDNet.

Downloads and confirmed runs
In 2021, we had more than 23,600 benchmark downloads and 228,900 confirmed runs. Our most popular benchmark, WebXPRT, just passed 909,800 runs since its debut in 2013! WebXPRT continues to be a go-to, industry-standard performance benchmark for OEM labs, vendors, and leading tech press outlets around the globe.

Media, publications, and interactive tools
Part of our mission with the XPRTs is to produce tools and materials that help testers better understand the ins and outs of benchmarking in general and the XPRTs in particular. To help achieve this goal, we published the following in 2021:

We’re thankful for everyone who has used the XPRTs, joined the community, and sent questions and suggestions throughout 2021. We look forward to an exciting 2022!

Justin

Posted in Battery life, battery life, benchmark, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, CloudXPRT, Collaborative benchmark development, CrXPRT, data analytics, results, WebXPRT, webxprt 4 | Tagged , , , , , , , |

A huge milestone for XPRT runs and downloads!

Justin Greene
By Justin Greene
on July 1, 2021

We’re excited to have recently passed an important milestone: one million XPRT runs and downloads! Most importantly, that huge number does not just reflect past successes. As the chart below illustrates, XPRT use has grown steadily over the years. In 2021, we record, on average, more XPRT runs and downloads in one month (23,395) than we recorded in the entire first year we started tracking these stats (17,051).

We reached one million runs and downloads in about seven and a half years. At the current rate, we’ll reach two million in roughly three and a half more years. With WebXPRT 4 on the way, there’s a good chance we can reach that mark even sooner!

As always, we’re grateful for all the testers that have helped us reach this milestone. If you have any questions or comments about using any of the XPRTs to test your gear, let us know!

Justin

Posted in benchmark, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, History of benchmarking, WebXPRT, webxprt 4 | Tagged , , , , , |

The CloudXPRT v1.1 beta is on the way

Justin Greene
By Justin Greene
on February 11, 2021

As we’ve been working on improvements and updates for CloudXPRT, we’ve been using feedback from community members to determine which changes will help testers most in the short term. To make some of those changes available to the community as soon as possible, we plan to release a beta version of CloudXPRT v1.1 in the coming weeks.

During the v1.1 beta period, the CloudXPRT v1.01 installation packages on CloudXPRT.com and our GitHub repository will continue to include the officially supported version of CloudXPRT. However, interested testers can experiment with the v1.1 beta version in new environments while we finalize the build for official release. 

The CloudXPRT v1.1 beta includes the following primary changes:

  • We’re adding support for Ubuntu 20.04.2 or later, the number one request we’ve received.
  • We’re consolidating and standardizing the installation packages for both workloads. Instead of one package for the data analytics workload and four separate packages for the web microservices workload, each workload will have two installation packages: one for all on-premises testing and one for testing with all three supported CSPs.
  • We’re incorporating Terraform to help create and configure VMs, which will help to prevent situations when testers do not allocate enough storage per VM prior to testing.
  • We use Kubespray to manage Kubernetes clusters, and Kubespray uses Calico as the default network plug in. Calico has not always worked well for CloudXPRT in the CSP environment, so we’re replacing Calico with Weave.


At the start of the beta period, we will share a link to the v1.1 beta download page here in the blog. You’ll be free to share this link. To avoid confusion, we will not add the beta download to the v1.01 downloads available on CloudXPRT.com.

As the beta release date approaches, we’ll share more details about timelines, access, and any additional changes to the benchmark. If you have any questions about the upcoming CloudXPRT v1.1 beta, please let us know!

Justin

Posted in benchmark, Benchmark metrics, BenchmarkXPRT, BenchmarkXPRT development community, Cloud, CloudXPRT, Clusters, Collaborative benchmark development, Cross-platform benchmarks, data analytics, Datacenter, GitHub, Hosted cloud, IaaS, Kubernetes, Linux, on premises, Performance benchmarking, web microservices | Tagged , , , , , , , , , |

Improved CloudXPRT documentation is coming soon

Justin Greene
By Justin Greene
on January 21, 2021

CloudXPRT is undoubtedly the most complex tool in the XPRT family of benchmarks. To run the cloud-native benchmark’s multiple workloads across different hardware and software platforms, testers need two things: (1) at least a passing familiarity with a wide range of cloud-related toolkits, and (2) an understanding that changing even one test configuration variable can affect test results. While the complexity of CloudXPRT makes it a powerful and flexible tool for measuring application performance on real-world IaaS stacks, it also creates a steep learning curve for new users.

Benchmark setup and configuration can involve a number of complex steps, and the corresponding instructions should be thorough, unambiguous, and intuitive to follow. For all of the XPRT tools, we strive to publish documentation that provides quick, easy-to-find answers to the questions users might have. Community members have asked us to improve the clarity and readability of the CloudXPRT setup, configuration, and individual workload documentation. In response, we are working to create more—and better—CloudXPRT documentation.

If you’re intimidated by the benchmark’s complexity, helping you is one of our highest priorities. In the coming weeks and months, we’ll be evaluating all of our CloudXPRT documentation, particularly from the perspective of new users, and will release more information about the new documentation as it becomes available.

We also want to remind you of some of the existing CloudXPRT resources. We encourage everyone to check out the Introduction to CloudXPRT and Overview of the CloudXPRT Web Microservices Workload white papers. (Note that we’ll soon be publishing a paper on the benchmark’s data analytics workload.) Also, a couple of weeks ago, we published the CloudXPRT learning tool, which we designed to serve as an information hub for common CloudXPRT topics and questions, and to help tech journalists, OEM lab engineers, and everyone who is interested in CloudXPRT find the answers they need as quickly as possible.

Thanks to all who let us know that there was room for improvement in the CloudXPRT documentation. We rely on that kind of feedback and always welcome it. If you have any questions or suggestions regarding CloudXPRT or any of the other XPRTs, please let us know!

Justin

Posted in benchmark, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Cloud, CloudXPRT, Cross-platform benchmarks, Hosted cloud, Hybrid cloud, IaaS, microservices, on premises, Performance benchmarking, White papers | Tagged , , , , , |

Following up

Justin Greene
By Justin Greene
on October 1, 2020

This week, we’re sharing news on two topics that we’ve discussed here in the blog over the past several months: CloudXPRT v1.01 and a potential AIXPRT OpenVINO update.

CloudXPRT v1.01

Last week, we announced that we were very close to releasing an updated CloudXPRT build (v1.01) with two minor bug fixes, an improved post-test results processing script, and an adjustment to one of our test configuration recommendations. Our testing and prep is complete, and the new version is live in the CloudXPRT GitHub repository and on our site!

None of the v1.01 changes affect performance or test results, so scores from the new build are comparable to those from previous CloudXPRT builds. If you’d like to know more about the changes, take a look at last week’s blog post.

The AIXPRT OpenVINO update

In late July, we discussed our plans to update the AIXPRT OpenVINO packages with OpenVINO 2020.3 Long-Term Support (LTS). While there are no known problems with the existing AIXPRT OpenVINO package, the LTS version targets environments that benefit from maximum stability and don’t require a constant stream of new tools and feature changes, so we thought it would be well suited for a benchmark like AIXPRT.

We initially believed that the update process would be relatively simple, and we’d be able to release a new AIXPRT OpenVINO package in September. However, we’ve discovered that the process is involved enough to require substantial low-level recoding. At this time, it’s difficult to estimate when the updated build will be ready for release. For any testers looking forward to the update, we apologize for the delay.

If you have any questions or comments about these or any other XPRT-related topics, please let us know!

Justin

Posted in AI, AIXPRT, benchmark, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Cloud, CloudXPRT, Datacenter, GitHub, Machine learning, OpenVINO, Performance benchmarking | Tagged , , , , , |

WebXPRT 3: relevant, reliable, and easy to use

Justin Greene
By Justin Greene
on September 17, 2020

WebXPRT continues to be the most widely-used XPRT benchmark, with just over 625,000 runs to date. From the first WebXPRT release in 2013, WebXPRT has been popular with device manufacturers, developers, tech journalists, and consumers because it’s easy to run, it runs on almost anything with a web browser, and its workloads reflect the types of web-based tasks that people are likely to encounter on a daily basis.

We realize that many folks who follow the XPRTs may be unaware of the wide variety of WebXPRT uses that we frequently read about in the tech press. Today, we thought it would be interesting to bring the numbers to life. In addition to dozens of device reviews, here’s a sample of WebXPRT 3 mentions over the past few weeks.

As we plan for the next version of WebXPRT, we want to be sure we build a benchmark that continues WebXPRT’s legacy of relevant workloads, ease-of-use, and broad compatibility. We know what works well in our lab, but to build a benchmark that meets the needs of a diverse group of users all around the world, it’s important that we hear from all types of testers. We recently discussed some of the new technologies that we’re considering for WebXPRT 4, so please don’t hesitate to let us know what you think about those proposals, or send any additional ideas you may have!

Justin

Posted in AnandTech, benchmark, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Browser-based benchmarks, Performance benchmarking, Web-based testing, WebXPRT, WebXPRT 3 | Tagged , , , , , , |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2022   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x