In recent blog posts, we discussed
an issue that we encountered when attempting to run WebXPRT 4 on iOS 17 devices.
If you missed those posts, you can find more details about the nature of the
problem here. In
short, the issue is that the Encrypt Notes and OCR scan subtest in WebXPRT 4
gets stuck when the Tesseract.js Optical Character Recognition (OCR) engine
attempts to scan a shopping receipt. We’ve verified that the issue occurs on
devices running iOS 17, iPadOS 17, and macOS Sonoma with Safari 17.
After a good bit of troubleshooting and research to try and identify the cause of the problem, we decided to build an updated version of WebXPRT 4 that uses a newer version of Tesseract for the OCR task. Aside from updating Tesseract in the new build, we aimed to change as little as possible. To try and maximize continuity, we’re still using the original input image for the receipt scanning task, and we decided to stick with using the WASM library instead of a WASM-SIMD library. Aside from a new version of tesseract.js, WebXPRT 4 version number updates, and updated documentation where necessary, all other aspects of WebXPRT 4 will remain the same.
testing a candidate build of this new version on a wide array of devices. The
results so far seem promising, but we want to complete our due diligence and
make sure this is the best approach to solving the problem. We know that OEM
labs and tech reviewers put a lot of time and effort into compiling databases
of results, so we hope to provide a solution that minimizes results disruption
and inconvenience for WebXPRT 4 users. Ideally, folks would be able to
integrate scores from the new build without any questions or confusion about comparability.
We don’t yet have an exact release date for a new WebXPRT 4 build, but we can say that we’re shooting for the end of October. We appreciate everyone’s patience as we work towards the best possible solution. If you have any questions or concerns about an updated version of WebXPRT 4, please let us know.
As part of our commitment to publishing reliable, unbiased benchmarks, we strive to make the XPRT development process as transparent as possible. In the technology assessment industry, it’s not unusual for people to claim that any given benchmark contains hidden biases, so we take preemptive steps to address this issue by publishing XPRT benchmark source code, detailed system disclosures and test methodologies, and in-depth white papers. Today, we’re focusing on the XPRT white paper library.
XPRT white paper library currently contains 21 white papers that we’ve
published over the last 12 years. We started publishing white papers to provide
XPRT users with more information about how we design our benchmarks, why we
make certain development decisions, and how the benchmarks work. If you have
questions about any aspect of one of the XPRT benchmarks, the white paper
library is a great place to find some answers.
example, the Exploring WebXPRT 4 white
paper describes the design and structure of WebXPRT 4, including detailed
information about the benchmark’s harness, HTML5 and WebAssembly (WASM)
capability checks, and the structure of the performance test workloads. It also
includes explanations of the benchmark’s scoring methodology, how to automate
tests, and how to submit results for publication.
companion WebXPRT 4 results calculation white paper explains the formulas that
WebXPRT 4 uses to calculate the individual workload scenario scores and overall
score, provides an overview of the statistical techniques WebXPRT uses to
translate raw timings into scores, and explains the benchmark’s confidence
interval and how it differs from typical benchmark variability. To supplement
the white paper’s discussion of the results calculation process, we published
a results calculation spreadsheet that shows the raw data from a sample test run and
reproduces the exact calculations WebXPRT uses to produce test scores.
hope that the XPRT white paper library will prove to be a useful resource for
you. If you have questions about any of our white papers, or suggestions for
topics that you’d like us to cover in possible future white papers, please let us know!
We’re excited to
announce that it’s been 10 years since the initial launch of WebXPRT! In early
2013, we introduced WebXPRT as a unique browser performance benchmark in a market
space that was already crowded with a variety of specialized measurement tools.
Our goal was to offer a benchmark that could compare the performance of almost
any web-enabled device, using scenarios created to mirror real-world tasks. We
wanted it to be a free, easily accessible, easy-to-run, useful, and appealing
testing option for OEM labs, vendors, and the tech press.
When we look back on
the last 10 years of WebXPRT, we can’t help but conclude that our efforts have
been successful. Since those early days, the WebXPRT market presence has grown
from humble beginnings into a worldwide industry standard. Hundreds of tech
press publications have used WebXPRT in thousands of articles and reviews, and testers
have now run the benchmark well over 1.1 million times.
Below, I’ve listed
some of the WebXPRT team’s accomplishments over the last decade. If you’ve been
following WebXPRT from the beginning, this may all be familiar, but if you’re
new to the community, it may be
interesting to see some of the steps that contributed to making WebXPRT what it
In future blog posts, we’ll look at how the number of WebXPRT runs has grown over time, and how WebXPRT use has grown among OEMs, vendors, and the tech press worldwide. Do you have any thoughts that you’d like to share from your WebXPRT testing experience? If so, let us know!
This week, we published the Exploring WebXPRT 4 white paper. It describes the design and structure of WebXPRT 4, including detailed information about the benchmark’s harness, HTML5 and WebAssembly (WASM) capability checks, and changes we’ve made to the structure of the performance test workloads. We explain the benchmark’s scoring methodology, how to automate tests, and how to submit results for publication. The white paper also includes information about the third-party functions and libraries that WebXPRT 4 uses during the HTML5 and WASM capability checks and performance workloads.
The Exploring WebXPRT 4 white paper promotes
the high level of transparency and disclosure that is a core value of the
BenchmarkXPRT Development Community. We’ve always believed that transparency
builds trust, and trust is essential for a healthy benchmarking community.
That’s why we involve community members in the benchmark development process
and disclose how we build our benchmarks and how they work.
You can find the paper on WebXPRT.com and our XPRT white papers page. If you have any questions about WebXPRT 4, please let us know, and be sure to check out our other XPRT white papers.
We’re excited to announce that WebXPRT 4 is now available! Testers can access the benchmark at WebXPRT.com. If you’ve already been using the WebXPRT 4 Preview, your Preview test results will be comparable with results from the current official build.
Longtime WebXPRT users will notice that WebXPRT 4 has a new, but familiar, UI. The general process for kicking off both manual and automated tests is the same as with WebXPRT 3, so the transition from WebXPRT 3 to WebXPRT 4 testing should be straightforward. We will continue to make WebXPRT 3 available for legacy testing.
If you missed
earlier XPRT blog posts
about WebXPRT 4, here is a quick overview of the differences between WebXPRT 3
and WebXPRT 4:
- We’ve updated the aesthetics of the WebXPRT UI to make WebXPRT 4 visually distinct from older versions. We did not significantly change the flow of the UI.
- We’ve updated content in some of the workloads to reflect changes in everyday technology, such as upgrading most of the photos in the photo processing workloads to higher resolutions.
- We’ve updated the base calibration system for score calculations, and adjusted the scoring scale. WebXPRT 4 scores should not be compared to scores from previous versions of WebXPRT.
- Photo Enhancement. We increased the efficiency of the workload’s Canvas object creation function, and replaced the existing photos with new, higher-resolution photos.
- Organize Album Using AI. We replaced ConvNetJS with WebAssembly (WASM) based OpenCV.js for both the face detection and image classification tasks. We changed the images for the image classification tasks to images from the ImageNet dataset.
- Stock Option Pricing. We updated the dygraph.js library.
- Sales Graphs. We made no changes to this workload.
- Encrypt Notes and OCR Scan. We replaced ASM.js with WASM for the Notes task and updated the WASM-based Tesseract version for the OCR task.
- Online Homework. In addition to the existing scenario which uses four Web Workers, we have added a scenario with two Web Workers. The workload now covers a wider range of Web Worker performance, and we calculate the score by using the combined run time of both scenarios. We also updated the typo.js library.
thankful for all of the feedback we received during the WebXPRT 4 development
process, and we look forward to seeing your WebXPRT 4 results!
The WebXPRT 4 development process is
far enough along that we’d like to share more about changes we are likely to
make and a rough target date for publishing a preview build. While some of the
details below will probably change, this post should give readers a good sense
of what to expect.
Some of the non-workload changes in
WebXPRT 4 relate to our typical benchmark update process, and a few result
directly from feedback we received from the WebXPRT tech press survey.
- We will update the aesthetics of the WebXPRT UI to make
WebXPRT 4 visually distinct from older versions. We do not anticipate
significantly changing the flow of the UI.
- We will update content in some of the workloads to
reflect changes in everyday technology. For instance, we will upgrade most
of the photos in the photo processing workloads to higher resolutions.
- In response to a request from tech press survey
respondents, we are considering adding a looping function to the
- We are investigating the possibility of shortening the
benchmark by reducing the default number of iterations from seven to five.
We will only make this change if we can ensure that five iterations produce
consistently low score variance.
Changes to existing workloads
Enhancement. This workload applies three effects
to two photos each (six photos total). It tests HTML5 Canvas, Canvas 2D, and
- Organize Album Using AI. This workload currently uses the ConvNetJS neural network library to complete two tasks: (1) organizing five images and (2) classifying the five images in an album. We are planning to replace ConvNetJS with WebAssembly (WASM) for both tasks and are considering upgrading the images to higher resolutions.
- Stock Option Pricing. This workload calculates and displays graphic views of a stock portfolio using Canvas, SVG, and dygraph.js. The only change we are considering is combining it with the Sales Graphs workload (below).
- Sales Graphs. This workload provides a web-based application displaying multiple views of sales data. Sales Graphs exercises HTML5 Canvas and SVG performance. The only change we are considering is combining it with the Stock Option Pricing workload (above).
- Encrypt Notes and OCR Scan. This workload uses ASM.js to sync notes, extract text from a scanned receipt using optical character recognition (OCR), and add the scanned text to a spending report. We are planning to replace ASM.js with WASM for the Notes task and with WASM-based Tesseract for the OCR task.
- Online Homework. This workload uses regex, arrays, strings, and Web Workers to review DNA and spell-check an essay. We are not planning to change this workload.
Possible new workloads
- Natural Language Processing (NLP). We are considering the addition of an NLP workload using ONNX Runtime and/or TensorFlowJS. The workload would use Bidirectional Encoder Representations from Transformers (BERT) to answer questions about a given text. Similar use cases are becoming more prevalent in conversational bot systems, domain-specific document search tools, and various other educational applications.
- Message Scrolling. We are considering developing a new workload that would use an Angular or React.js to scroll through hundreds of messages. We’ll share more about this possible workload as we firm up the details.
The release timeline
We hope to publish a WebXPRT 4
preview build in the second half of November, with a general release before the
end of the year. If it looks as though that timeline will change significantly,
we’ll provide an update here in the blog as soon as possible.
We’re very grateful for all the
input we received during the WebXPRT 4 planning process. If you have any
questions about the details we’ve shared above, please feel free to ask!