Earlier this month, we
discussed a few of our ideas for possible changes in WebXPRT 4, including
new web technologies that may work well in a browser benchmark. Today, we’re
going to focus on one of those technologies, WebAssembly, in more detail.
(WASM) is a binary instruction format that works across all modern browsers. WASM
provides a sandboxed environment that operates at native speeds and takes
advantage of common hardware specs across platforms. WASM’s capabilities offer
web developers a great deal of flexibility for running complex client
applications in the browser. That level of flexibility may enable workload scenario
options for WebXPRT 4 such as
gaming, video editing, VR, virtual machines, and image recognition. We’re
excited about those possibilities, but it remains to be seen which WASM use
cases will meet the criteria we look for when considering new WebXPRT
workloads, such as relevancy to real life, consistency and replicability, and
the broadest-possible level of cross-browser support.
One WASM workload that
we’re investigating is a web-based machine learning workload with TensorFlow
tasks, including image classification, object detection, sentence encoding, and
natural language processing. TensorFlow.js originally used WebGL technology on
the back end, but now it’s possible to run the workload using WASM. We could
also use this technology to enhance one of WebXPRT’s existing AI-themed
workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.
We’re can’t yet say that a WASM workload will definitely appear in WebXPRT 4, but the technology is promising. Do you have any experience with WASM, or ideas for WASM workloads? There’s still time for you to help shape the future of WebXPRT 4, so let us know what you think!
CrXPRT testers may
remember that back around the time that we began the CrXPRT 2 development process, the Chrome team announced that they were
phasing out support for Portable Native Client (PNaCL) in favor of WebAssembly (WASM). As a first step,
they changed the Chrome OS setting that enabled PNaCL by default. At the time,
this caused problems with the Photo Collage workload in CrXPRT 2015, and even
though we identified a workaround, details in the Chrome team’s announcement led us to conclude
that the workaround might stop working in June 2021. Because of this change, we
decided that the best decision would be to remove the workload from CrXPRT
2, and keep existing CrXPRT 2015 testers informed of any changes with the
In 2020, the Chrome
team also announced that they would be phasing out support for Chrome Apps
altogether starting in June 2021, and would shift their focus to Chrome
extensions. This change would have required us to reassess the viability of
CrXPRT in anything like its current form.
We’re happy to report that
the Chrome team has extended support for PNaCL and existing Chrome Apps through
June 2022. Barring further changes, this means that CrXPRT
2015 (with the workaround) and CrXPRT 2 should continue to serve as reliable
Chrome OS evaluation tools for some time.
If you have any questions about CrXPRT 2, please let us know!
We’re happy to announce
that CloudXPRT v1.1 will move from beta to general release status tomorrow! The
installation packages will be available at the CloudXPRT.com download page and the BenchmarkXPRT GitHub repository. You will find more details about the v1.1
updates in a previous blog post, but the most
prominent changes are the consolidation of the five previous installation
packages into two packages (one per workload) and added support for Ubuntu
20.04.2 with on-premises testing.
Before you get started
with v1.1, please note the following updated system requirements:
- Ubuntu 20.04.2 or later for on-premises testing
- Ubuntu 18.04 and 20.04.2 or later for CSP (AWS/Azure/GCP)
CloudXPRT is designed
to run on high-end servers. Physical nodes or VMs under test must meet the
following minimum specifications:
- 16 logical or virtual CPUs
- 8 GB of RAM
- 10 GB of available disk space (50 GB for the data analytics
We have also made
significant adjustments to the installation and test configuration instructions
in the readmes for both workloads, so please revisit these documents even if
you’re familiar with previous test processes.
As we noted during the
beta period, we have not observed any significant differences in performance
between v1.01 and v1.1, but we haven’t tested every possible test configuration
across every platform. If you observe different results when testing the same
configuration with v1.01 and v1.1, please send us the details so we can
If you have any questions about CloudXPRT v1.1, please let us know!
In the coming months,
we’ll be moving forward with the first stages of the WebXPRT 4 development
process. It’s been a while since we last asked readers to send their
thoughts about web technologies and workloads that may be a good fit for
WebXPRT 4, but we’re still very much open to ideas. If you missed our previous
posts about possible changes for WebXPRT 4, we recap the most prominent ideas
below. We also request specific feedback regarding a potential battery life
- Community members have asked about a WebXPRT 4 battery life test. Any such test would likely be very similar to the performance-weighted battery life test in CrXPRT 2 (as opposed to a simple rundown test). While WebXPRT runs in almost any browser, cross-browser compatibility issues could cause a WebXPRT battery life test to run in only one browser. If this turned out to be the case, would you still be interested in using the battery life test? Please let us know.
- One of the most promising ideas is the potential addition of one or more WebAssembly (WASM) workloads. WASM is a low-level, binary instruction format that works across all modern browsers. It offers web developers a great deal of flexibility and provides the speed and efficiency necessary for running complex client applications in the browser. WASM enables a variety of workload scenario options, including gaming, video editing, VR, virtual machines, image recognition, and interactive educational content.
- Other ideas include using a WebGL-based workload to target GPUs, and simulating common web applications.
We’ll start work on
WebXPRT 4 soon, but there’s still time to send your comments and ideas, so please
do so as quickly as possible!
Last week, we announced that a CloudXPRT v1.1
beta was on the way. We’re happy to say that the v1.1 beta is now available to
the public on a dedicated CloudXPRT v1.1 beta download page. While CloudXPRT v1.01
remains the officially supported version on CloudXPRT.com and in our GitHub
repository, interested testers can use the v1.1
beta version in new environments as we finalize the v1.1 build for official
release. You are welcome to publish results as we do not expect results to
change in the final, official release.
As we mentioned in
last week’s post, the CloudXPRT v1.1 beta includes the following changes:
- We’ve added support for Ubuntu 20.04.2 or later for on-premises
- We’ve consolidated and standardized the installation packages
for both workloads. Instead of one package for the data analytics workload and
four separate packages for the web microservices workload, each workload has a
single installation package that supports on-premises testing and testing with
all three supported CSPs.
- We’ve incorporated Terraform to help create and
configure VMs, which helps to prevent problems when testers do not allocate
enough storage per VM prior to testing.
- We’ve replaced the Calico network plugin in Kubespray with Weave, which helps to avoid some
of the network issues testers have occasionally encountered in the CPS
Please feel free to
share the link to the beta download page. (To avoid confusion, the beta will
not appear in the main CloudXPRT download table.) We can’t yet state
definitively whether results from the new version will be comparable to those
from v1.01. We have not observed any significant differences in performance,
but we haven’t tested every possible test configuration across every platform.
If you observe different results when testing the same configuration with v1.01
and v1.1 beta, please send us the details so we can investigate.
If you have any questions about CloudXPRT or the CloudXPRT v1.1 beta, please let us know!
As we’ve been working
on improvements and updates for CloudXPRT, we’ve been using feedback from
community members to determine which changes will help testers most in the
short term. To make some of those changes available to the community as soon as
possible, we plan to release a beta version of CloudXPRT v1.1 in the coming
During the v1.1 beta
period, the CloudXPRT v1.01 installation packages on CloudXPRT.com and our GitHub repository will continue to include the officially supported
version of CloudXPRT. However, interested testers can experiment with the v1.1
beta version in new environments while we finalize the build for official
The CloudXPRT v1.1
beta includes the following primary changes:
- We’re adding support for Ubuntu 20.04.2 or later, the number one
request we’ve received.
- We’re consolidating and standardizing the installation packages
for both workloads. Instead of one package for the data analytics workload and
four separate packages for the web microservices workload, each workload will
have two installation packages: one for all on-premises testing and one for
testing with all three supported CSPs.
- We’re incorporating Terraform to help create and
configure VMs, which will help to prevent situations when testers do not
allocate enough storage per VM prior to testing.
- We use Kubespray to manage Kubernetes
clusters, and Kubespray uses Calico as the default network plug in. Calico has not always worked
well for CloudXPRT in the CSP environment, so we’re replacing Calico with Weave.
At the start of the
beta period, we will share a link to the v1.1 beta download page here in the
blog. You’ll be free to share this link. To avoid confusion, we will not add the
beta download to the v1.01 downloads available on CloudXPRT.com.
As the beta release
date approaches, we’ll share more details about timelines, access, and any additional
changes to the benchmark. If you have any questions about the upcoming
CloudXPRT v1.1 beta, please let us know!