We recently received questions about whether we accept CloudXPRT
results submissions from testing on pre-production gear, and how we would handle
any differences between results from pre-production and production-level tests.
To answer first question, we are not opposed to pre-production
results submissions. We realize that vendors often want to include benchmark
results in launch-oriented marketing materials they release before their
hardware or software is publicly available. To help them do so, we’re happy to
consider pre-production submissions on a case-by-case basis. All such submissions
must follow the normal CloudXPRT results
submission process, and undergo
vetting by the CloudXPRT Results Review Group according to the standard review
and publication schedule. If we decide to publish pre-production results on our site, we
will clearly note their pre-production status.
In response to the second question, the CloudXPRT Results Review Group will handle any challenges to published results or perceived discrepancies between pre-production and production-level results on a case-by-case basis. We do not currently have a formal process for challenges; anyone who would like to initiate a challenge or express comments or concerns about a result should address the review group via firstname.lastname@example.org. Our primary concern is always to ensure that published results accurately reflect the performance characteristics of production-level hardware and software. If it becomes necessary to develop more policies in the future, we’ll do so, but we want to keep things as simple as possible.
If you have any questions about the CloudXPRT results submission process, please let us know!
In recent lab tests, we’ve encountered an error during the CrXPRT 2 battery life test that prevents the test from completing and producing a battery life estimate. As the screenshot below shows, when the error occurs, CrXPRT stops running its normal workload cycle and produces a “Test Error” page. We have seen this behavior on systems running Chrome OS v89.x and v90.x, across multiple vendor platforms. In our testing, Chromebooks running Chrome OS v88.x and earlier versions continue to complete the battery life test without any issues.
The error occurs consistently on every Chromebook running v89.x or
v90.x that we’ve tested so far. However, the timing of the error varies from
run to run on the same system. Sometimes, CrXPRT stops running after only a few
workload iterations, while at other times, the battery life test runs almost to
completion before producing the error.
We’re actively investigating this problem, but have not yet
identified the root cause. We apologize for the inconvenience that this error
may be causing CrXPRT 2 testers. As soon as we identify the root cause of the
problem and have ideas about possible solutions, we will share that information
here in the blog. If you have any insight into recent Chrome OS changes or flag
settings that could be causing this problem, please let us know!
Device reviews in publications
such as AnandTech, Notebookcheck, and PCMag, among many others, often feature
WebXPRT test results, and we appreciate the many members of the tech press that
use WebXPRT. As we move forward with the WebXPRT 4 development process, we’re especially
interested in learning what longtime users would like to see in a new version
of the benchmark.
In previous posts,
we’ve asked people to weigh in on the potential addition of a WebAssembly workload or a battery life test. We’d also like to ask experienced testers some other
test-related questions. To that end, this week we’ll be sending a WebXPRT 4
survey directly to members of the tech press who frequently publish WebXPRT
Regardless of whether you are a member of the tech press, we invite you to participate by sending your answers to any or all the questions below to email@example.com. We ask you to do so by the end of May.
Do you think WebXPRT 3’s selection of workload scenarios is representative of modern web tasks?
How do you think WebXPRT compares to other common browser-based benchmarks, such as JetStream, Speedometer, and Octane?
Are there web technologies that you’d like us to include in additional workloads?
Are you happy with the WebXPRT 3 user interface? If not, what UI changes would you like to see?
Are there any aspects of WebXPRT 2015 that we changed in WebXPRT 3 that you’d like to see us change back?
Have you ever experienced significant connection issues when testing with WebXPRT?
Given the array of workloads, do you think the WebXPRT runtime is reasonable? Would you mind if the average runtime were a bit longer?
Are there any other aspects of WebXPRT 3 that you’d like to see us change?
If you’d like to discuss any topics
that we did not cover in the questions above, please feel free to include additional
comments in your response. We look forward to hearing your thoughts!
We’re excited to see
that users have successfully completed over 750,000 WebXPRT runs! If you’ve run WebXPRT in any of the more than 654 cities
and 68 countries from which we’ve received complete test data—including
newcomers Belize, Cambodia, Croatia, and Pakistan—we’re grateful for your help.
We could not have reached this milestone without you!
As the chart below illustrates, WebXPRT use has grown steadily over the years. We now record, on average, almost twice as many WebXPRT runs in one month as we recorded in the entirety of our first year. In addition, with over 82,000 runs to date in 2021, there are no signs that growth is slowing.
Developing a new
benchmark is never easy, and the obstacles multiply when you attempt to create
a cross-platform benchmark, such as WebXPRT, that will run on a wide variety of
devices. Establishing trust with the benchmarking community is another
challenge. Transparency, consistency, and technical competency on our part are critical
factors in building that trust, but the people who take time out of their busy
schedules to run the benchmark for the first time also play a role. We thank
all of the manufacturers, OEM labs, and members of the tech press who decided
to give WebXPRT a try, and we look forward to your input as we continue to improve WebXPRT in the years to come.
If you have any
questions or comments about WebXPRT, we’d love to hear from you!
Earlier this month, we
discussed a few of our ideas for possible changes in WebXPRT 4, including
new web technologies that may work well in a browser benchmark. Today, we’re
going to focus on one of those technologies, WebAssembly, in more detail.
(WASM) is a binary instruction format that works across all modern browsers. WASM
provides a sandboxed environment that operates at native speeds and takes
advantage of common hardware specs across platforms. WASM’s capabilities offer
web developers a great deal of flexibility for running complex client
applications in the browser. That level of flexibility may enable workload scenario
options for WebXPRT 4 such as
gaming, video editing, VR, virtual machines, and image recognition. We’re
excited about those possibilities, but it remains to be seen which WASM use
cases will meet the criteria we look for when considering new WebXPRT
workloads, such as relevancy to real life, consistency and replicability, and
the broadest-possible level of cross-browser support.
One WASM workload that
we’re investigating is a web-based machine learning workload with TensorFlow
tasks, including image classification, object detection, sentence encoding, and
natural language processing. TensorFlow.js originally used WebGL technology on
the back end, but now it’s possible to run the workload using WASM. We could
also use this technology to enhance one of WebXPRT’s existing AI-themed
workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.
We’re can’t yet say that a WASM workload will definitely appear in WebXPRT 4, but the technology is promising. Do you have any experience with WASM, or ideas for WASM workloads? There’s still time for you to help shape the future of WebXPRT 4, so let us know what you think!
CloudXPRT is undoubtedly
the most complex tool in the XPRT family of benchmarks. To run the cloud-native
benchmark’s multiple workloads across different hardware and software platforms,
testers need two things: (1) at least a passing familiarity with a wide range
of cloud-related toolkits, and (2) an understanding that changing even one test
configuration variable can affect test results. While the complexity of CloudXPRT
makes it a powerful and flexible tool for measuring application performance on
real-world IaaS stacks, it also creates a steep learning curve for new users.
Benchmark setup and
configuration can involve a number of complex steps, and the corresponding
instructions should be thorough, unambiguous, and intuitive to follow. For all
of the XPRT tools, we strive to publish documentation that provides quick,
easy-to-find answers to the questions users might have. Community members have asked
us to improve the clarity and readability of the CloudXPRT setup,
configuration, and individual workload documentation. In response, we are
working to create more—and better—CloudXPRT documentation.
If you’re intimidated
by the benchmark’s complexity, helping you is one of our highest priorities. In
the coming weeks and months, we’ll be evaluating all of our CloudXPRT
documentation, particularly from the perspective of new users, and will release
more information about the new documentation as it becomes available.
We also want to remind
you of some of the existing CloudXPRT resources. We encourage everyone to check
out the Introduction to CloudXPRT and Overview of the CloudXPRT Web Microservices Workload white papers. (Note
that we’ll soon be publishing a paper on the benchmark’s data analytics
workload.) Also, a couple of weeks ago, we published the CloudXPRT learning tool, which we designed to serve as an information
hub for common CloudXPRT topics and questions, and to help tech journalists,
OEM lab engineers, and everyone who is interested in CloudXPRT find the answers
they need as quickly as possible.
Thanks to all who let us know that there was room for improvement in the CloudXPRT documentation. We rely on that kind of feedback and always welcome it. If you have any questions or suggestions regarding CloudXPRT or any of the other XPRTs, please let us know!