A few months ago, we invited readers to send in their thoughts and ideas about web
technologies and workload scenarios that may be a good fit for the next WebXPRT. We’d like to share a few of those ideas today, and we invite
you to continue to send your feedback. We’re approaching the time when we need to begin firming up
plans for a WebXPRT 4 development cycle in 2021, but there’s still plenty of
time for you to help shape the future of the benchmark.
One of the most
promising ideas for WebXPRT 4 is the potential addition of one or more WebAssembly (WASM) workloads.
WASM is a low-level, binary instruction format that works across all modern browsers.
It offers web developers a great deal of flexibility and provides the speed and
efficiency necessary for running complex client applications in the browser. WASM
enables a variety of workload scenario options, including gaming, video editing, VR, virtual
machines, image recognition, and interactive educational content.
In addition, the
Chrome team is dropping Portable Native Client (PNaCL) support in favor of
WASM, which is why we had to remove a PNaCL workload when updating CrXPRT 2015 to CrXPRT 2. We
generally model CrXPRT workloads on existing WebXPRT workloads, so
familiarizing ourselves with WASM could ultimately benefit more than one XPRT
We are also
considering adding a web-based machine learning workload with TensorFlow for
tasks including image classification, object detection, sentence encoding,
natural language processing, and more. We could also use this technology to
enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album
using AI or Encrypt Notes and OCR Scan.
Other ideas include using
a WebGL-based workload to target GPUs and investigating ways to incorporate a
battery life test. What do you think? Let us know!
BenchmarkXPRT Development Community started almost 10 years ago with the development
of the High Definition Experience & Performance Ratings Test, also known as
HDXPRT. Back then, we distributed the benchmark to interested parties by
mailing out physical DVDs. We’ve come a long way since then, as testers now
freely and easily access six XPRT benchmarks from our site and major app
hardware manufacturers, and tech journalists—the core group of XPRT testers—work
within a constantly changing tech landscape. Because of our commitment to
providing those testers with what they need, the XPRTs grew as we developed
additional benchmarks to expand the reach of our tools from PCs to servers and
all types of notebooks, Chromebooks, and mobile devices.
today’s tech landscape continues to evolve at a rapid pace, our desire to play
an active role in emerging markets continues to drive us to expand our testing
capabilities into areas like machine learning (AIXPRT)
and cloud-first applications (CloudXPRT).
While these new technologies carry the potential to increase efficiency, improve
quality, and boost the bottom line for companies around the world, it’s often
difficult to decide where and how to invest in new hardware or services. The
ever-present need for relevant and reliable data is the reason many
organizations use the XPRTs to help make confident choices about their
company’s future tech.
We just released a new video that helps to explain what the XPRTs provide and how they can play an important role in a company’s tech purchasing decisions. We hope you’ll check it out!
excited about the continued growth of the XPRTs, and we’re eager to meet the
challenges of adapting to the changing tech landscape. If you have any questions
about the XPRTs or suggestions for future benchmarks, please let us know!
much of the BenchmarkXPRT Development Community’s history, we offered community
members exclusive access to XPRT benchmark source code. Back in February,
we started to experiment with a different approach when we made the AIXPRT
source code publicly available on GitHub. By allowing anyone who is
interested in AIXPRT to download and review the source code, we reinforced our
commitment to making the XPRT development process as transparent as possible. We
also want the XPRTs to continue to contribute to fair practices in the
benchmarking world, and we believe that expanded access to the source code
encourages constructive feedback to help in this goal.
feedback we received after publishing the AIXPRT source code was very positive;
thank you to all who reached out. Because of that feedback and our desire to
increase openness, we’ve decided use standard open source licenses to make the CloudXPRT
source code available to the public when we release of the first build, or
shortly thereafter. As with AIXPRT, folks will be able to download the CloudXPRT
source code and submit potential workloads for future consideration, but we
reserve the right to control derivative works.
share more information about the first CloudXPRT release and its source code in
the coming weeks. If you have any questions about XPRT source code, feel free to ask.
We also welcome any thoughts about using
this approach to release the source code of other XPRT benchmarks. As always, feel
free to comment below or reach out by email.
It’s been about two years since we released WebXPRT 3, and we’re starting to think about the WebXPRT 4 development cycle. With over 529,000 runs to date, WebXPRT continues to be our most popular benchmark because it’s quick and easy to run, it runs on almost anything with a web browser, and it evaluates performance using the types of web technologies that many people use every day.
For each new version of WebXPRT, we start the development process by looking at browser trends and analyzing the feasibility of incorporating new web technologies into our workload scenarios. For example, in WebXPRT 3, we updated the Organize Album workload to include an image-classification task that uses deep learning. We also added an optical character recognition task to the Encrypt Notes and OCR scan workload, and introduced a new Online Homework workload that combined part of the DNA Sequence Analysis scenario with a writing sample/spell check scenario.
Here are the current WebXPRT 3 workloads:
Photo Enhancement: Applies three effects, each using Canvas, to two photos.
Organize Album Using AI: Detects faces and classifies images using the ConvNetJS neural network library.
Stock Option Pricing: Calculates and displays graphic views of a stock portfolio using Canvas, SVG, and dygraphs.js.
Encrypt Notes and OCR Scan: Encrypts notes in local storage and scans a receipt using optical character recognition.
Sales Graphs: Calculates and displays multiple views of sales data using InfoVis and d3.js.
Online Homework: Performs science and English assignment tasks using Web Workers and Typo.js spell check.
What new technologies or workload scenarios should we add? Are there any existing features we should remove? Would you be interested in an associated battery life test? We want to hear your thoughts and ideas about WebXPRT, so please tell us what you think!
month, Bill announced
that we were starting work on a new data center benchmark. CloudXPRT
will measure the performance of modern, cloud-first applications deployed on infrastructure
as a service (IaaS) platforms—on-premises platforms,
externally hosted platforms, and hybrid clouds that use a mix of the two. Our
ultimate goal is for CloudXPRT to use cloud-native components on an actual
stack to produce end-to-end performance metrics that can help users determine the
right IaaS configuration for their business.
we want to provide a quick update on CloudXPRT development and testing.
Installation. We’ve completely automated the CloudXPRT installation process, which leverages Kubernetes or Ansible tools depending on the target platform. The installation processes differ slightly for each platform, but testing is the same.
Workloads. We’re currently testing potential workloads that focus on three areas: web microservices, data analytics, and container scaling. We might not include all of these workloads in the first release, but we’ll keep the community informed and share more details about each workload as the picture becomes clearer. We are designing the workloads so that testers can use them to directly compare IaaS stacks and evaluate whether any given stack can meet service level agreement (SLA) thresholds.
Platforms. We want CloudXPRT to eventually support testing on a variety of popular externally hosted platforms. However, constructing a cross-platform benchmark is complicated and we haven’t yet decided which external platforms the first CloudXPRT release will support. We’ve successfully tested the current build with on-premises IaaS stacks and with one externally hosted platform, Amazon Web Services. Next, we will test the build on Google Cloud Hosting and Microsoft Azure.
Timeline. We are on track to meet our target of releasing a CloudXPRT preview build in late March and the first official build about two months later. If anything changes, we’ll post an updated timeline here in the blog.
you would like to share any thoughts or comments related to CloudXPRT or cloud
benchmarking, please feel free to contact
This week, we
have good news for AIXPRT testers: the AIXPRT source code is now available to the public
via GitHub. As we’ve discussed in the past, publishing XPRT source code is part of our
commitment to making the XPRT development process as transparent as
possible. With other XPRT benchmarks, we’ve only made the source code available
to community members. With AIXPRT, we have released the source code more
widely. By allowing all interested parties, not just community members, to
download and review our source code, we’re taking tangible steps to improve
openness and honesty in the benchmarking industry and we’re encouraging the
kind of constructive feedback that helps to ensure that the XPRTs continue to
contribute to a level playing field.
Traditional open-source models encourage developers to change products and even take them in new and different directions. Because benchmarking requires a product that remains static to enable valid comparisons over time, we allow people to download the source code and submit potential workloads for future consideration, but we reserve the right to control derivative works. This discourages a situation where someone publishes an unauthorized version of the benchmark and calls it an “XPRT.”
We encourage you to download and review the source and send us any feedback you may have. Your questions and suggestions may influence future versions of AIXPRT. If you have any questions about AIXPRT or accessing the source code, please feel free to ask! Please also let us know if you think we should take this approach to releasing the source code with other XPRT benchmarks.