As we’ve been working
on improvements and updates for CloudXPRT, we’ve been using feedback from
community members to determine which changes will help testers most in the
short term. To make some of those changes available to the community as soon as
possible, we plan to release a beta version of CloudXPRT v1.1 in the coming
During the v1.1 beta
period, the CloudXPRT v1.01 installation packages on CloudXPRT.com and our GitHub repository will continue to include the officially supported
version of CloudXPRT. However, interested testers can experiment with the v1.1
beta version in new environments while we finalize the build for official
The CloudXPRT v1.1
beta includes the following primary changes:
We’re adding support for Ubuntu 20.04.2 or later, the number one
request we’ve received.
We’re consolidating and standardizing the installation packages
for both workloads. Instead of one package for the data analytics workload and
four separate packages for the web microservices workload, each workload will
have two installation packages: one for all on-premises testing and one for
testing with all three supported CSPs.
We’re incorporating Terraform to help create and
configure VMs, which will help to prevent situations when testers do not
allocate enough storage per VM prior to testing.
We use Kubespray to manage Kubernetes
clusters, and Kubespray uses Calico as the default network plug in. Calico has not always worked
well for CloudXPRT in the CSP environment, so we’re replacing Calico with Weave.
At the start of the
beta period, we will share a link to the v1.1 beta download page here in the
blog. You’ll be free to share this link. To avoid confusion, we will not add the
beta download to the v1.01 downloads available on CloudXPRT.com.
As the beta release
date approaches, we’ll share more details about timelines, access, and any additional
changes to the benchmark. If you have any questions about the upcoming
CloudXPRT v1.1 beta, please let us know!
Today, we expand our portfolio
of CloudXPRT resources with a paper on the benchmark’s data analytics workload.
While we summarized the workload in the Introduction to CloudXPRT white paper, the new paper goes into much
In addition to providing practical information about the data analytics installation package and minimum system requirements, the paper describes the workload’s test configuration variables, structural components, task workflows, and test metrics. It also discusses interpreting test results and the process for submitting results for publication.
CloudXPRT is the most
complex tool in the XPRT family, and the new paper is part of our effort to create more—and better—CloudXPRT documentation. We plan to
publish additional CloudXPRT white papers in the coming months, with possible
future topics including the impact of adjusting specific test configuration
options, recommendations for results reporting, and methods for analysis.
CloudXPRT is undoubtedly
the most complex tool in the XPRT family of benchmarks. To run the cloud-native
benchmark’s multiple workloads across different hardware and software platforms,
testers need two things: (1) at least a passing familiarity with a wide range
of cloud-related toolkits, and (2) an understanding that changing even one test
configuration variable can affect test results. While the complexity of CloudXPRT
makes it a powerful and flexible tool for measuring application performance on
real-world IaaS stacks, it also creates a steep learning curve for new users.
Benchmark setup and
configuration can involve a number of complex steps, and the corresponding
instructions should be thorough, unambiguous, and intuitive to follow. For all
of the XPRT tools, we strive to publish documentation that provides quick,
easy-to-find answers to the questions users might have. Community members have asked
us to improve the clarity and readability of the CloudXPRT setup,
configuration, and individual workload documentation. In response, we are
working to create more—and better—CloudXPRT documentation.
If you’re intimidated
by the benchmark’s complexity, helping you is one of our highest priorities. In
the coming weeks and months, we’ll be evaluating all of our CloudXPRT
documentation, particularly from the perspective of new users, and will release
more information about the new documentation as it becomes available.
We also want to remind
you of some of the existing CloudXPRT resources. We encourage everyone to check
out the Introduction to CloudXPRT and Overview of the CloudXPRT Web Microservices Workload white papers. (Note
that we’ll soon be publishing a paper on the benchmark’s data analytics
workload.) Also, a couple of weeks ago, we published the CloudXPRT learning tool, which we designed to serve as an information
hub for common CloudXPRT topics and questions, and to help tech journalists,
OEM lab engineers, and everyone who is interested in CloudXPRT find the answers
they need as quickly as possible.
Thanks to all who let us know that there was room for improvement in the CloudXPRT documentation. We rely on that kind of feedback and always welcome it. If you have any questions or suggestions regarding CloudXPRT or any of the other XPRTs, please let us know!
We’re happy to announce
that the CloudXPRT learning tool is now live! We
designed the tool to serve as an information hub for common CloudXPRT topics
and questions, and to help tech journalists, OEM lab engineers, and everyone
who is interested in CloudXPRT find the answers they need as quickly as
The tool features four
primary areas of content:
The Q&A section provides quick answers to the questions we
receive most from testers and the tech press.
The CloudXPRT: the basics section describes specific topics such
as the benchmark’s target platforms, workloads, companion cloud software, and
hardware and software requirements.
The Testing and results section covers the testing process,
metrics, and how to publish results.
The cloud primer provides brief, easy-to-understand definitions of
key cloud computing terms and concepts.
The first screenshot below shows the home screen. To illustrate how some of the pop-up information sections appear, the second screenshot shows part of the Key terms and concepts module in the Cloud primer section.
We’re excited about the new CloudXPRT learning tool! If you have any questions about the tool, or suggestions for additional content to include in it, please let us know!
This week, we’re sharing news on two topics that we’ve discussed
here in the blog over the past several months: CloudXPRT v1.01 and a potential
AIXPRT OpenVINO update.
Last week, we announced that we were very close to releasing an
updated CloudXPRT build (v1.01) with two minor bug fixes, an improved post-test
results processing script, and an adjustment to one of our test configuration
recommendations. Our testing and prep is complete, and the new version is live
in the CloudXPRT GitHub repository and on our site!
None of the v1.01
changes affect performance or test results, so scores from the new build are
comparable to those from previous CloudXPRT builds. If you’d like to know more
about the changes, take a look at last week’s blog post.
The AIXPRT OpenVINO
In late July, we discussed our plans to update the AIXPRT OpenVINO packages
with OpenVINO 2020.3 Long-Term Support (LTS). While there are no
known problems with the existing AIXPRT OpenVINO package, the LTS version
targets environments that benefit from maximum stability and don’t require a
constant stream of new tools and feature changes, so we thought it would be
well suited for a benchmark like AIXPRT.
We initially believed that
the update process would be relatively simple, and we’d be able to release a
new AIXPRT OpenVINO package in September. However, we’ve discovered that the
process is involved enough to require substantial low-level recoding. At this
time, it’s difficult to estimate when the updated build will be ready for
release. For any testers looking forward to the update, we apologize for the
If you have any questions or comments about
these or any other XPRT-related topics, please let us know!
We want to let CloudXPRT testers know that we’re close to
releasing an updated version (build 1.01) with two minor bug fixes, an improved
post-test results processing script, and an adjustment to one of our test
configuration recommendations. None of these changes will affect performance or
test results, so scores from previous CloudXPRT builds will be comparable to
those from the new build.
The most significant changes in CloudXPRT build 1.01 are as
In previous builds, some testers encountered warnings during setup to update the version of Kubernetes Operations (kops) when testing on public-cloud platforms (the CloudXPRT 1.00 recommendation is kops version 1.16.0). We are adjusing the kops installation instructions in the setup instructions for the web microservices and data analytics workloads to prevent these warnings.
In previous builds, post-test cleanup instructions for public-cloud testing environments do not always delete all of the resources that CloudXPRT creates during setup. We are updating instructions to ensure a more thorough cleanup process. This change applies to test instructions for the web microservices and data analytics workloads.
We are reformatting the optional results graphs the web microservices postprocess program creates to make them easier to interpret.
In previous builds, the recommended time interval for the web-microservices workload is 120 seconds if the hpamode option is enabled and 60 seconds if it is disabled. Because we’ve found that the 60-second difference has no significant impact on test results, we are changing the recommendation to 60 seconds for both hpamode settings.
We hope these changes
will improve the CloudXPRT setup and testing experience. We haven’t set the
release date for the updated build yet, but when we do, we’ll announce it here
in the blog. If you have any questions about CloudXPRT, or would like to report
bugs or other issues, please feel free to contact us!