recently published a set of CloudXPRT Data Analytics and Web Microservices
workload test results
submitted by Quanta Computer, Inc.
The Quanta submission is the first set of CloudXPRT results that we’ve
published using the formal results submission and approval process.
We’re grateful to the Quanta team for carefully following the submission
guidelines, enabling us to complete the review process without a hitch.
If you are unfamiliar
with the process, you can find general information about how we review
submissions in a previous blog post.
Detailed, step-by-step instructions are available on the results submission page.
As a reminder for testers who are considering submitting results for July, the
submission deadline is tomorrow, Friday July 16, and the publication date is
Friday July 30. We list the submission and publication dates for the rest of
2021 below. Please note that we do not plan to review submissions in December,
so if we receive results submissions after November 30, we may not publish them
until the end of January 2022.
Submission deadline: Tuesday 8/17/21
Publication date: Tuesday 8/31/21
Submission deadline: Thursday 9/16/21
Publication date: Thursday 9/30/21
Submission deadline: Friday 10/15/21
Publication date: Friday 10/29/21
Submission deadline: Tuesday 11/16/21
Publication date: Tuesday 11/30/21
Submission deadline: N/A
Publication date: N/A
If you have any questions about the CloudXPRT results submission, review, or publication process, please let us know!
We recently received questions about whether we accept CloudXPRT
results submissions from testing on pre-production gear, and how we would handle
any differences between results from pre-production and production-level tests.
To answer first question, we are not opposed to pre-production
results submissions. We realize that vendors often want to include benchmark
results in launch-oriented marketing materials they release before their
hardware or software is publicly available. To help them do so, we’re happy to
consider pre-production submissions on a case-by-case basis. All such submissions
must follow the normal CloudXPRT results
submission process, and undergo
vetting by the CloudXPRT Results Review Group according to the standard review
and publication schedule. If we decide to publish pre-production results on our site, we
will clearly note their pre-production status.
In response to the second question, the CloudXPRT Results Review Group will handle any challenges to published results or perceived discrepancies between pre-production and production-level results on a case-by-case basis. We do not currently have a formal process for challenges; anyone who would like to initiate a challenge or express comments or concerns about a result should address the review group via firstname.lastname@example.org. Our primary concern is always to ensure that published results accurately reflect the performance characteristics of production-level hardware and software. If it becomes necessary to develop more policies in the future, we’ll do so, but we want to keep things as simple as possible.
If you have any questions about the CloudXPRT results submission process, please let us know!
We’re happy to announce
that CloudXPRT v1.1 will move from beta to general release status tomorrow! The
installation packages will be available at the CloudXPRT.com download page and the BenchmarkXPRT GitHub repository. You will find more details about the v1.1
updates in a previous blog post, but the most
prominent changes are the consolidation of the five previous installation
packages into two packages (one per workload) and added support for Ubuntu
20.04.2 with on-premises testing.
Before you get started
with v1.1, please note the following updated system requirements:
- Ubuntu 20.04.2 or later for on-premises testing
- Ubuntu 18.04 and 20.04.2 or later for CSP (AWS/Azure/GCP)
CloudXPRT is designed
to run on high-end servers. Physical nodes or VMs under test must meet the
following minimum specifications:
- 16 logical or virtual CPUs
- 8 GB of RAM
- 10 GB of available disk space (50 GB for the data analytics
We have also made
significant adjustments to the installation and test configuration instructions
in the readmes for both workloads, so please revisit these documents even if
you’re familiar with previous test processes.
As we noted during the
beta period, we have not observed any significant differences in performance
between v1.01 and v1.1, but we haven’t tested every possible test configuration
across every platform. If you observe different results when testing the same
configuration with v1.01 and v1.1, please send us the details so we can
If you have any questions about CloudXPRT v1.1, please let us know!
Last week, we announced that a CloudXPRT v1.1
beta was on the way. We’re happy to say that the v1.1 beta is now available to
the public on a dedicated CloudXPRT v1.1 beta download page. While CloudXPRT v1.01
remains the officially supported version on CloudXPRT.com and in our GitHub
repository, interested testers can use the v1.1
beta version in new environments as we finalize the v1.1 build for official
release. You are welcome to publish results as we do not expect results to
change in the final, official release.
As we mentioned in
last week’s post, the CloudXPRT v1.1 beta includes the following changes:
- We’ve added support for Ubuntu 20.04.2 or later for on-premises
- We’ve consolidated and standardized the installation packages
for both workloads. Instead of one package for the data analytics workload and
four separate packages for the web microservices workload, each workload has a
single installation package that supports on-premises testing and testing with
all three supported CSPs.
- We’ve incorporated Terraform to help create and
configure VMs, which helps to prevent problems when testers do not allocate
enough storage per VM prior to testing.
- We’ve replaced the Calico network plugin in Kubespray with Weave, which helps to avoid some
of the network issues testers have occasionally encountered in the CPS
Please feel free to
share the link to the beta download page. (To avoid confusion, the beta will
not appear in the main CloudXPRT download table.) We can’t yet state
definitively whether results from the new version will be comparable to those
from v1.01. We have not observed any significant differences in performance,
but we haven’t tested every possible test configuration across every platform.
If you observe different results when testing the same configuration with v1.01
and v1.1 beta, please send us the details so we can investigate.
If you have any questions about CloudXPRT or the CloudXPRT v1.1 beta, please let us know!
Soon, we’ll be expanding
our portfolio of CloudXPRT resources with a white paper that focuses on the benchmark’s
web microservices workload. While we summarized the workload in the Introduction to CloudXPRT white paper, the new paper will discuss the
workload in much greater detail.
In addition to providing practical information about the web microservices installation packages and minimum system requirements, the paper describes the workload’s test configuration variables, structural components, task workflows, and test metrics. It also discusses interpreting test results and the process for submitting results for publication.
As we’ve noted, CloudXPRT is one of the more complex tools in the XPRT family, with no shortage of topics to explore further. We plan to publish a companion overview for the data analytics workload, and possible future topics include the impact of adjusting specific test configuration options, recommendations for results reporting, and methods for analysis.
We hope that the
upcoming Overview of the CloudXPRT Web Microservices Workload paper will
serve as a go-to resource for CloudXPRT testers, and will answer any questions
you have about the workload. Once it goes live, we’ll provide links in the
Helpful Info box on CloudXPRT.com and the CloudXPRT section of our XPRT white papers page.
If you have any questions,
please let us know!
The CloudXPRT Preview period has ended, and CloudXPRT version 1.0 installation packages are now available on CloudXPRT.com and the BenchmarkXPRT GitHub repository! Like the Preview build, CloudXPRT version 1.0 includes two workloads: web microservices and data analytics (you can find more details about the workloads here). Testers can use metrics from the workloads to compare IaaS stack (both hardware and software) performance and to evaluate whether any given stack is capable of meeting SLA thresholds. You can configure CloudXPRT to run on local datacenter, Amazon Web Services, Google Cloud Platform, or Microsoft Azure deployments.
Several different test packages are available for download from the CloudXPRT download page. For detailed installation instructions and hardware and software requirements for each, click the package’s readme link. On CloudXPRT.com, the Helpful Info box contains resources such as links to the Introduction to CloudXPRT white paper, the CloudXPRT master readme, and the CloudXPRT GitHub repository.
The GitHub repository also contains the CloudXPRT
source code. The source code is freely available for testers to download and
Performance results from this release are comparable
to performance results from the CloudXPRT Preview build. Testers who wish to
publish results on CloudXPRT.com can find more information about the results
submission and review process in the blog. We post the monthly results cycle schedule on the results
We’re thankful for all the input we received during the CloudXPRT development process and Preview period. If you have any questions about CloudXPRT, please let us know.