BenchmarkXPRT Blog banner

Category: Cross-platform benchmarks

The CloudXPRT v1.2 update package is now available!

Justin Greene
By Justin Greene
on October 27, 2022

We’re happy to announce that the CloudXPRT v1.2 update package is now available! The update prevents potential installation failures on Google Cloud Platform and Microsoft Azure, and ensures that the web microservices workload works on Ubuntu 22.04. The update uses updated software components such as Kubernetes v1.23.7, Kubespray v2.18.1, and Kubernetes Metrics Server v1, and incorporates some additional minor script changes.

The CloudXPRT v1.2 web microservices workload installation package is available at the CloudXPRT.com download page and the BenchmarkXPRT GitHub repository.

Before you get started with v1.2, please note the following updated system requirements:

  • Ubuntu 20.04.2 or 22.04 for on-premises testing
  • Ubuntu 18.04, 20.04.2, or 22.04 for CSP (AWS/Azure/GCP) testing

Because CloudXPRT is designed to run on high-end servers, physical nodes or VMs under test must meet the following minimum specifications:

  • 16 logical or virtual CPUs
  • 8 GB of RAM
  • 10 GB of available disk space (50 GB for the data analytics workload)

The update package includes only the updated v1.2 test harness and the updated web microservices workload. It does not include the data analytics workload. As we stated in the blog, now that we’ve published the web microservices package, we will assess the level of interest users express about a possible refresh of the v1.1 data analytics workload. For now, the v1.1 data analytics workload will continue to be available via CloudXPRT.com for some time to serve as a reference resource for users who have worked with the package in the past.

Please let us know if you have any questions about the CloudXPRT v1.2 test package. Happy testing!

Justin

Posted in Amazon Web Services, benchmark, Benchmark metrics, Cloud, CloudXPRT, Cross-platform benchmarks, CSP, data analytics, Datacenter, Google Cloud Platform, Kubernetes, microservices, Microsoft Azure, On-premise, Performance benchmarking, Servers, Ubuntu, web microservices | Tagged , , , , , , , , , , , , |

The versatility of XPRT benchmarks

Justin Greene
By Justin Greene
on October 13, 2022

We’ve designed each of the XPRT benchmarks to assess the performance of specific types of devices in scenarios that mirror the ways consumers typically use those devices. While most XPRT benchmark users are interested in producing official overall scores, some members of the tech press have been using the XPRTs in unconventional, creative ways.

One example is the use of WebXPRT by Tweakers, a popular tech review site based in The Netherlands. (The site is in Dutch, so the Google Translate extension in Chrome was helpful for me.) As Tweakers uses WebXPRT to evaluate all kinds of consumer hardware, they also measure the sound output of each device. Tweakers then publishes the LAeq metric for each device, giving readers a sense of how loud a system may be, on average, while it performs common browser tasks.

If you’re interested in seeing Tweakers’ use of WebXPRT for sound output testing firsthand, check out their Apple MacBook Pro M2, HP Envy 34 All-in-One, and Samsung Galaxy Book 2 Pro reviews.

Other labs and tech publications have also used the XPRTs in unusual ways such as automating the benchmarks to run during screen burn-in tests or custom battery-life rundowns. If you’ve used any of the XPRT benchmarks in creative ways, please let us know! We are interested in learning more about your tests, and your experiences may provide helpful information that we can share with other XPRT users.

Justin

Posted in Apple, Benchmark metrics, Benchmarking, Browser-based benchmarks, Cross-platform benchmarks, Google Chrome, Performance benchmarking, Web-based testing, WebXPRT, WebXPRT 3, WebXPRT 4 | Tagged , , , , , , , , , |

On track for a CloudXPRT web microservices update this fall

Justin Greene
By Justin Greene
on September 29, 2022

Last month, we announced that we’re working on an updated CloudXPRT web microservices test package. The purpose of the update is to fix installation failures on Google Cloud Platform and Microsoft Azure, and ensure that the web microservices workload works on Ubuntu 22.04, using updated software components such as Kubernetes v1.23.7, Kubespray v2.18.1, and Kubernetes Metrics Server v1. The update also incorporates some additional minor script changes.

We are still testing the updated test package with on-premises hardware and Amazon Web Services, Google Cloud Platform, and Microsoft Azure configurations. So far, testing is progressing well, and we feel increasingly confident that we will be able to release the updated test package soon. We would like to share a more concrete release schedule, but because of the complexity of the workload and the CSP platforms involved, we are waiting until we are certain that everything is ready to go.

The name of the updated package will be CloudXPRT v1.2, and it will include only the updated v1.2 test harness and the updated web microservices workload. It will not include the data analytics workload. As we stated in last month’s blog, we plan to publish the updated web microservices package, and see what kind of interest we receive from users about a possible refresh of the v1.1 data analytics workload. For now, the v1.1 data analytics workload will continue to be available via CloudXPRT.com for some time to serve as a reference resource for users that have worked with the package in the past.

As soon as possible, we’ll provide more information about the CloudXPRT v1.2 release date here in the blog. If you have any questions about the update or CloudXPRT in general, please feel free to contact us!

Justin

Posted in Amazon Web Services, benchmark, Cloud, CloudXPRT, Cross-platform benchmarks, Google Cloud Platform, Hosted cloud, Hybrid cloud, IaaS, Kubernetes, Microsoft Azure, on premises, Performance benchmarking | Tagged , , , , , , , , , |

CloudXPRT status and next steps

Justin Greene
By Justin Greene
on August 11, 2022

We developed our first cloud benchmark, CloudXPRT, to measure the performance of cloud applications deployed on modern infrastructure as a service (IaaS) platforms. When we first released CloudXPRT in February of 2021, the benchmark included two test packages: a web microservices workload and a data analytics workload. Both supported on-premises and cloud service provider (CSP) testing with Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure. 

CloudXPRT is our most complex benchmark, requiring sustained compatibility between many software components across multiple independent test environments. As vendors roll out updates for some components and stop supporting others, it’s inevitable that something will break. Since CloudXPRT’s launch, we’ve become aware of installation failures while attempting to set up CloudXPRT on Ubuntu virtual machines with GCP and Microsoft Azure. Additionally, while the web microservices workload continues to run in most instances with a few configuration tweaks and workarounds, the data analytics workload fails consistently due to compatibility issues with Minio, Prometheus, and Kafka within the Kubernetes environment. 

In response, we’re working to fix problems with the web microservices workload and bring all necessary components up to date. We’re developing an updated test package that will work on Ubuntu 22.04, using Kubernetes v1.23.7 and Kubespray v2.18.1. We’re also updating Kubernetes Metrics Server from v1beta1 to v1, and will incorporate some minor script changes. Our goal is to ensure successful installation and testing with the on-premises and CSP platforms that we supported when we first launched CloudXPRT.

We are currently focusing on the web microservices workload for two reasons. First, more users have downloaded it than the data analytics workload. Second, we think we have a clear path to success. Our plan is to publish the updated web microservices test package, and see what feedback and interest we receive from users about a possible data analytics refresh. The existing data analytics workload will remain available via CloudXPRT.com for the time being to serve as a reference resource.

We apologize for the inconvenience that these issues have caused. We’ll provide more information about a release timeline and final test package details here in the blog as we get closer to publication. If you have any questions about the future of CloudXPRT, please feel free to contact us!

Justin

Posted in Amazon Web Services, benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Cloud, CloudXPRT, Cross-platform benchmarks, CSP, data analytics, Datacenter, Google Cloud Platform, Hosted cloud, Hybrid cloud, IaaS, Kubernetes, microservices, Microsoft Azure, On-premise, Performance benchmarking, web microservices | Tagged , , , , , , , , , , |

Helpful tips for WebXPRT 4 results submission

Justin Greene
By Justin Greene
on June 9, 2022

Back in March, we discussed the WebXPRT 4 results submission process and reminded readers that everyone who runs a WebXPRT 4 test is welcome to submit scores for us to consider for publication in the WebXPRT 4 results viewer. Unlike sites that publish every result that users submit, we publish only results that meet our evaluation criteria. Among other things, scores must be consistent with general expectations and must include enough detailed system information to help us assess whether individual scores represent valid test runs. Today, we offer a couple of tips to increase the likelihood that we will publish your WebXPRT 4 test results.

Tip 1: Specify your system’s processor

While testers usually include detailed information for the device, model number, operating system, and browser version fields, we receive many submissions with little to no information about the test system’s processor.

In the picture below, you can see an example of the level of detail that we require to consider a submission. We need the full processor name, including the manufacturer and model number (e.g., Intel Core i9-9980HK, AMD Ryzen 3 1300X, or Apple M1 Max). Note that we do not require the processor speed reported by the system.

Tip 2: Include a valid email address

It is also common for submissions to not include a valid email address. While we understand the privacy concerns related to submitting a personal or corporate email address, we need a valid address that we can use as a point of contact to confirm test-related information when necessary. We don’t use those addresses for any other purposes, such as selling them, sharing them with any third parties, or adding them to a mailing list.

We hope this information explains why we might not have published your results. We look forward to receiving your future score submissions. If you have any questions about the submission process, please let us know!

Justin

Posted in AMD, Apple, benchmark, Benchmark metrics, Browser-based benchmarks, Cross-platform benchmarks, Intel, Performance benchmarking, results, results submission, WebXPRT, WebXPRT 4 | Tagged , , , , , |

XPRTs in the press

Justin Greene
By Justin Greene
on May 12, 2022

Each month, we send a newsletter to members of the BenchmarkXPRT Development Community. In the newsletter, we recap the latest updates from the XPRT world and provide a summary of the previous month’s XPRT-related activity, including uses or mentions of the XPRTs in the tech press. More people read the weekly XPRT blog than receive the monthly newsletter, so we realized that some blog readers may be unaware of the wide variety of tech outlets that regularly use or mention the XPRTs.

So for today’s blog, we want to give readers a sampling of the XPRT press usage we see on a weekly basis. Recent mentions include:

  • Tom’s Guide used HDXPRT 4 to compare the performance of the Geekom Mini IT8 and Dell OptiPlex 7090 Ultra small-form-factor PCs.
  • Intel used WebXPRT 4 test data in promotional material for their line of 12th Gen) Intel Core processors(Alder Lake). Hundreds of press outlets then republished the presentation.
  • AnandTech used WebXPRT 4 to evaluate the Cincoze DS-1300 Industrial PC.
  • ZDNet used CrXPRT 2 in a review titled The best Chromebooks for students: Student-proof laptops.
  • PCWorld used CrXPRT 2 to provide data for an article listing their top Chromebook recommendations.
  • TechPowerUp used WebXPRT 3 to compare the browser performance of Intel Core i9-12900KS processor-based systems and other Intel- and AMD processor-based systems.
  • Other outlets that have published articles, ads, or reviews mentioning the experts in the last few months include: Android Authority, ASUS, BenchLife, Gadgets 360, Good Gear Guide, Hardware.info, Hot Hardware, ITHardware (Poland), ITMedia (Japan), Itndaily (Russia), Mobile01.com (China), Notebookcheck, PCMag, ProClockers, Sohu.com (China), Tom’s Hardware, and Tweakers.

If you don’t currently receive the monthly BenchmarkXPRT newsletter, but would like to join the mailing list, please let us know! We will not publish or sell any of the contact information you provide, and will only send the monthly newsletter and occasional benchmark-related announcements such as patch notifications or new benchmark releases.

Justin

Posted in benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Chromebooks, Cross-platform benchmarks, CrXPRT, HDXPRT, HDXPRT 4, Performance benchmarking, Performance of computing devices, WebXPRT, WebXPRT 3, webxprt 4 | Tagged , , , , , , , , , , |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2024   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x