Last week, we
published the Exploring WebXPRT 4 white paper.
The paper describes the design and structure of WebXPRT 4, including detailed
information about the benchmark’s harness, HTML5 and WebAssembly capability
checks, and the structure of the performance test workloads. This week, to
help WebXPRT 4 testers understand how the benchmark calculates results, we’ve published
the WebXPRT 4 results calculation and confidence interval white
paper explains the WebXPRT 4 confidence interval and how it differs from typical
benchmark variability, and the formulas the benchmark uses to calculate the
individual workload scenario scores and overall score. The paper also provides
an overview of the statistical techniques WebXPRT uses to translate raw timings
the white paper’s discussion of the results calculation process, we’ve also
published a results calculation spreadsheet that shows the
raw data from a sample test run and reproduces the calculations WebXPRT uses to
produce workload scores and the overall score.
The paper is available on WebXPRT.com and on our XPRT white papers page. If you have any questions about the WebXPRT results calculation process, please let us know!
This week, we published the Exploring WebXPRT 4 white paper. It describes the design and structure of WebXPRT 4, including detailed information about the benchmark’s harness, HTML5 and WebAssembly (WASM) capability checks, and changes we’ve made to the structure of the performance test workloads. We explain the benchmark’s scoring methodology, how to automate tests, and how to submit results for publication. The white paper also includes information about the third-party functions and libraries that WebXPRT 4 uses during the HTML5 and WASM capability checks and performance workloads.
The Exploring WebXPRT 4 white paper promotes
the high level of transparency and disclosure that is a core value of the
BenchmarkXPRT Development Community. We’ve always believed that transparency
builds trust, and trust is essential for a healthy benchmarking community.
That’s why we involve community members in the benchmark development process
and disclose how we build our benchmarks and how they work.
You can find the paper on WebXPRT.com and our XPRT white papers page. If you have any questions about WebXPRT 4, please let us know, and be sure to check out our other XPRT white papers.
some recent internal WebXPRT 4 Preview
testing, we discovered that the WebXPRT 4 Preview does not run in Internet
Explorer (IE) 11. In fact, before the first workload begins in IE, the WebXPRT
4 built-in WebAssembly (WASM) check fails and produces an error message.
reason we haven’t tested WebXPRT 4 on IE 11 before now is that Internet
Explorer is currently in its end-of-life phase. Microsoft has been removing
support for IE 11 in Microsoft 365 and other apps for some time, they did not
include the desktop version of IE 11 in Windows 11, and they are removing support for IE 11
in Windows 10 on June 15, 2022. Among Windows users, the most popular browsers
are now Microsoft Edge, Google Chrome, and Mozilla Firefox.
proud that WebXPRT has historically had broad, cross-platform compatibility in
almost any browser. However, the modern web is rapidly incorporating powerful
tools such as WASM that do not work in older legacy browsers. To maintain the
benchmark’s relevance in future years, we need to deprioritize some level of
legacy compatibility, and this begins with WebXPRT 4 release.
the WebXPRT testers who wish to continue testing with IE 11, WebXPRT 3 will remain
on our site for the foreseeable future. Barring any further changes from
Microsoft, the benchmark should continue to run in existing instances of the Internet
Explorer desktop app.
The official WebXPRT 4 launch is approaching, and we hope to announce the release date within the next few weeks! Until that time, we will continue to share the latest updates here in the blog. If you have any questions or comments about WebXPRT 4 or compatibility with legacy browsers, please feel free to contact us!
In October, we shared an early preview of the new results viewer tool that we’ve been developing in parallel with WebXPRT 4. The WebXPRT 4 Preview is now available to the public, and we’re excited to announce that the new results viewer is also live. We already have over 65 test results in the viewer, and in the weeks leading up to the WebXPRT 4 general release, we’ll be actively populating the viewer with the latest PT-curated WebXPRT 4 Preview results.
encourage readers to visit the blog for details
about the viewer’s features, and to take some time to explore the data.
We’re excited about this new tool, which we view as an ongoing project with
room for expansion and improvement based on user feedback.
If you have any questions or comments about the WebXPRT 4 Preview or the new results viewer, please feel free to contact us!
A few months ago, we shared detailed information about the changes we expected
to make in WebXPRT 4. We are currently doing internal testing of the WebXPRT 4 Preview
build in preparation for releasing it to the public. We want to let our readers
know what to expect.
We’ve made some changes since our
last update and some of the details we present below could still change before
the preview release. However, we are much closer to the final product. Once we
release the WebXPRT 4 Preview, testers will be able to publish scores from Preview
build testing. We will limit any changes that we make between the Preview and
the final release to the UI or features that are not expected to affect test
Some of the non-workload changes we’ve
made in WebXPRT 4 relate to our typical benchmark update process.
- We have updated the aesthetics of the WebXPRT UI to make WebXPRT 4 visually distinct from older versions. We did not significantly change the flow of the UI.
- We have updated content in some of the workloads to reflect changes in everyday technology, such as upgrading most of the photos in the photo processing workloads to higher resolutions.
- We have not yet added a looping function to the automation scripts, but are still considering it for the future.
- We investigated the possibility of shortening the benchmark by reducing the default number of iterations from seven to five, but have decided to stick with seven iterations to ensure that score variability remains acceptable across all platforms.
Enhancement. We increased the efficiency of the
workload’s Canvas object creation function, and replaced the existing photos
with new, higher-resolution photos.
- Organize Album Using AI. We replaced ConvNetJS with WebAssembly (WASM) based OpenCV.js for both the face detection and image classification tasks. We changed the images for the image classification tasks to images from the ImageNet dataset.
- Stock Option Pricing. We updated the dygraph.js library.
- Sales Graphs. We made no changes to this workload.
- Encrypt Notes and OCR Scan. We replaced ASM.js with WASM for the Notes task and updated the WASM-based Tesseract version for the OCR task.
- Online Homework. In addition to the existing scenario which uses four Web Workers, we have added a scenario with two Web Workers. The workload now covers a wider range of Web Worker performance, and we calculate the score by using the combined run time of both scenarios. We also updated the typo.js library.
As part of the WebXPRT 4 development
process, we researched the possibility of including two new workloads: a
natural language processing (NLP) workload, and an Angular-based message
scrolling workload. After much testing and discussion, we have decided to not
include these two workloads in WebXPRT 4. They will be good candidates for us
to add as experimental WebXPRT 4 workloads in 2022.
The release timeline
Our goal is to publish the WebXPRT 4
preview build by December 15th, which will allow testers to publish
scores in the weeks leading up to the Consumer Electronics Show in Las Vegas in
January 2022. We will provide more detailed information about the GA timeline
here in the blog as soon as possible.
If you have any questions about the details we’ve shared above, please feel free to ask!
The WebXPRT 4 development process is
far enough along that we’d like to share more about changes we are likely to
make and a rough target date for publishing a preview build. While some of the
details below will probably change, this post should give readers a good sense
of what to expect.
Some of the non-workload changes in
WebXPRT 4 relate to our typical benchmark update process, and a few result
directly from feedback we received from the WebXPRT tech press survey.
- We will update the aesthetics of the WebXPRT UI to make
WebXPRT 4 visually distinct from older versions. We do not anticipate
significantly changing the flow of the UI.
- We will update content in some of the workloads to
reflect changes in everyday technology. For instance, we will upgrade most
of the photos in the photo processing workloads to higher resolutions.
- In response to a request from tech press survey
respondents, we are considering adding a looping function to the
- We are investigating the possibility of shortening the
benchmark by reducing the default number of iterations from seven to five.
We will only make this change if we can ensure that five iterations produce
consistently low score variance.
Changes to existing workloads
Enhancement. This workload applies three effects
to two photos each (six photos total). It tests HTML5 Canvas, Canvas 2D, and
- Organize Album Using AI. This workload currently uses the ConvNetJS neural network library to complete two tasks: (1) organizing five images and (2) classifying the five images in an album. We are planning to replace ConvNetJS with WebAssembly (WASM) for both tasks and are considering upgrading the images to higher resolutions.
- Stock Option Pricing. This workload calculates and displays graphic views of a stock portfolio using Canvas, SVG, and dygraph.js. The only change we are considering is combining it with the Sales Graphs workload (below).
- Sales Graphs. This workload provides a web-based application displaying multiple views of sales data. Sales Graphs exercises HTML5 Canvas and SVG performance. The only change we are considering is combining it with the Stock Option Pricing workload (above).
- Encrypt Notes and OCR Scan. This workload uses ASM.js to sync notes, extract text from a scanned receipt using optical character recognition (OCR), and add the scanned text to a spending report. We are planning to replace ASM.js with WASM for the Notes task and with WASM-based Tesseract for the OCR task.
- Online Homework. This workload uses regex, arrays, strings, and Web Workers to review DNA and spell-check an essay. We are not planning to change this workload.
Possible new workloads
- Natural Language Processing (NLP). We are considering the addition of an NLP workload using ONNX Runtime and/or TensorFlowJS. The workload would use Bidirectional Encoder Representations from Transformers (BERT) to answer questions about a given text. Similar use cases are becoming more prevalent in conversational bot systems, domain-specific document search tools, and various other educational applications.
- Message Scrolling. We are considering developing a new workload that would use an Angular or React.js to scroll through hundreds of messages. We’ll share more about this possible workload as we firm up the details.
The release timeline
We hope to publish a WebXPRT 4
preview build in the second half of November, with a general release before the
end of the year. If it looks as though that timeline will change significantly,
we’ll provide an update here in the blog as soon as possible.
We’re very grateful for all the
input we received during the WebXPRT 4 planning process. If you have any
questions about the details we’ve shared above, please feel free to ask!