In recent blog posts, we discussed
an issue that we encountered when attempting to run WebXPRT 4 on iOS 17 devices.
If you missed those posts, you can find more details about the nature of the
problem here. In
short, the issue is that the Encrypt Notes and OCR scan subtest in WebXPRT 4
gets stuck when the Tesseract.js Optical Character Recognition (OCR) engine
attempts to scan a shopping receipt. We’ve verified that the issue occurs on
devices running iOS 17, iPadOS 17, and macOS Sonoma with Safari 17.
After a good bit of troubleshooting and research to try and identify the cause of the problem, we decided to build an updated version of WebXPRT 4 that uses a newer version of Tesseract for the OCR task. Aside from updating Tesseract in the new build, we aimed to change as little as possible. To try and maximize continuity, we’re still using the original input image for the receipt scanning task, and we decided to stick with using the WASM library instead of a WASM-SIMD library. Aside from a new version of tesseract.js, WebXPRT 4 version number updates, and updated documentation where necessary, all other aspects of WebXPRT 4 will remain the same.
testing a candidate build of this new version on a wide array of devices. The
results so far seem promising, but we want to complete our due diligence and
make sure this is the best approach to solving the problem. We know that OEM
labs and tech reviewers put a lot of time and effort into compiling databases
of results, so we hope to provide a solution that minimizes results disruption
and inconvenience for WebXPRT 4 users. Ideally, folks would be able to
integrate scores from the new build without any questions or confusion about comparability.
We don’t yet have an exact release date for a new WebXPRT 4 build, but we can say that we’re shooting for the end of October. We appreciate everyone’s patience as we work towards the best possible solution. If you have any questions or concerns about an updated version of WebXPRT 4, please let us know.
Recent visitors to CrXPRT.com may have seen a notice that encourages visitors to use WebXPRT 4
instead of CrXPRT 2 for performance testing on high-end Chromebooks. The notice
reads as follows:
Chromebook technology has progressed rapidly since we released CrXPRT 2, and
we’ve received reports that some CrXPRT 2 workloads may not stress top-bin
Chromebook processors enough to give the necessary accuracy for users to
compare their performance. So, for the latest test to compare the performance
of high-end Chromebooks, we recommend using WebXPRT 4.
made this recommendation because of the evident limitations of the CrXPRT 2
performance workloads when testing newer high-end hardware. CrXPRT 2 itself is
not that old (2020), but when we created the CrXPRT 2 performance workloads, we
started with a core framework of CrXPRT 2015 performance workloads. In a
similar way, we built the CrXPRT 2015 workloads on a foundation of WebXPRT 2015
workloads. At the time, the harness and workload structures we used to ensure
WebXPRT 2015’s cross-browser capabilities provided an excellent foundation that
we could adapt for our new ChromeOS benchmark. Consequently, CrXPRT 2 is a close
developmental descendant of WebXPRT 2015. Some of the legacy WebXPRT
2015/CrXPRT 2 workloads do not stress current high-end processors—a limitation that
prevents effective performance testing differentiation—nor do they engage the
latest web technologies.
the past, the Chromebook market skewed heavily toward low-cost devices with down-bin,
inexpensive processors, making this limitation less of an issue. Now, however,
more Chromebooks offer top-bin processors on par with traditional laptops and
workstations. Because of the limitations of the CrXPRT 2 workloads, we now recommend
WebXPRT 4 for both cross-browser and ChromeOS performance testing on the latest
tools and libraries, modern WebAssembly workloads, and additional Web Workers
tasks that cover a wide range of performance requirements.
CrXPRT 2 continues to function as a capable performance and battery life
comparison test for many ChromeOS devices, WebXPRT 4 is a more appropriate tool
to use with new high-end devices. If you haven’t yet used WebXPRT 4 for
Chromebook comparison testing, we encourage you to give it a try!
If you have any questions or concerns about CrXPRT 2 or WebXPRT 4, please don’t hesitate to ask!
As part of our commitment to publishing reliable, unbiased benchmarks, we strive to make the XPRT development process as transparent as possible. In the technology assessment industry, it’s not unusual for people to claim that any given benchmark contains hidden biases, so we take preemptive steps to address this issue by publishing XPRT benchmark source code, detailed system disclosures and test methodologies, and in-depth white papers. Today, we’re focusing on the XPRT white paper library.
XPRT white paper library currently contains 21 white papers that we’ve
published over the last 12 years. We started publishing white papers to provide
XPRT users with more information about how we design our benchmarks, why we
make certain development decisions, and how the benchmarks work. If you have
questions about any aspect of one of the XPRT benchmarks, the white paper
library is a great place to find some answers.
example, the Exploring WebXPRT 4 white
paper describes the design and structure of WebXPRT 4, including detailed
information about the benchmark’s harness, HTML5 and WebAssembly (WASM)
capability checks, and the structure of the performance test workloads. It also
includes explanations of the benchmark’s scoring methodology, how to automate
tests, and how to submit results for publication.
companion WebXPRT 4 results calculation white paper explains the formulas that
WebXPRT 4 uses to calculate the individual workload scenario scores and overall
score, provides an overview of the statistical techniques WebXPRT uses to
translate raw timings into scores, and explains the benchmark’s confidence
interval and how it differs from typical benchmark variability. To supplement
the white paper’s discussion of the results calculation process, we published
a results calculation spreadsheet that shows the raw data from a sample test run and
reproduces the exact calculations WebXPRT uses to produce test scores.
hope that the XPRT white paper library will prove to be a useful resource for
you. If you have questions about any of our white papers, or suggestions for
topics that you’d like us to cover in possible future white papers, please let us know!
We’re excited to
announce that it’s been 10 years since the initial launch of WebXPRT! In early
2013, we introduced WebXPRT as a unique browser performance benchmark in a market
space that was already crowded with a variety of specialized measurement tools.
Our goal was to offer a benchmark that could compare the performance of almost
any web-enabled device, using scenarios created to mirror real-world tasks. We
wanted it to be a free, easily accessible, easy-to-run, useful, and appealing
testing option for OEM labs, vendors, and the tech press.
When we look back on
the last 10 years of WebXPRT, we can’t help but conclude that our efforts have
been successful. Since those early days, the WebXPRT market presence has grown
from humble beginnings into a worldwide industry standard. Hundreds of tech
press publications have used WebXPRT in thousands of articles and reviews, and testers
have now run the benchmark well over 1.1 million times.
Below, I’ve listed
some of the WebXPRT team’s accomplishments over the last decade. If you’ve been
following WebXPRT from the beginning, this may all be familiar, but if you’re
new to the community, it may be
interesting to see some of the steps that contributed to making WebXPRT what it
In future blog posts, we’ll look at how the number of WebXPRT runs has grown over time, and how WebXPRT use has grown among OEMs, vendors, and the tech press worldwide. Do you have any thoughts that you’d like to share from your WebXPRT testing experience? If so, let us know!
As the number
of WebXPRT runs continues to grow, we realize many new WebXPRT users may be unfamiliar with all the
features and capabilities of the benchmark. To help inform users about features
that might facilitate their testing, we’ve decided to highlight a few WebXPRT
features here in the blog. A few weeks ago, we discussed the multiple language options available in the WebXPRT 4 UI.
This week, we look at WebXPRT 4 test automation.
WebXPRT 4 allows
users to run scripts in an automated fashion. You can control the execution of
WebXPRT 4 by appending parameters and values to the WebXPRT URL. Three
parameters are available: testtype, tests, and result. Below, you’ll find a
description of those parameters and instructions for utilizing automation.
automation framework accounts for two test types: (1) the six core workloads
and (2) any experimental workloads we might add in future builds. There are
currently no experimental tests in WebXPRT 4, so always set the test type
variable to 1.
parameter lets you specify which tests to run by using the following codes:
- Photo enhancement: 1
- Organize album using AI: 2
- Stock option pricing: 4
- Encrypt notes and OCR scan using WASM: 8
- Sales graphs: 16
- Online homework: 32
To run a single individual test, use its code. To run multiple
tests, use the sum of their codes. For example, to run Stocks (4) and Notes
(8), use the sum of 12. To run all core tests, use 63, the sum of all the
individual test codes (1 + 2 + 4 + 8 + 16 + 32 = 63).
lets you select the format of the results:
- Display the result as an HTML table: 1
- Display the result as XML: 2
- Display the result as CSV: 3
- Download the result as CSV: 4
To use the automation feature, start with the URL http://www.principledtechnologies.com/benchmarkxprt/webxprt/2021/wx4_build_3_7_3, append a
question mark (?), and add the parameters and values separated by ampersands
(&). For example, to run all the core tests and download the results, you
would use the following URL: http://principledtechnologies.com/benchmarkxprt/webxprt/2021/wx4_build_3_7_3/auto.php?testtype=1&tests=63&result=4
We hope the WebXPRT automation features will make testing easier
for you. If you have any questions about WebXPRT or the automation process,
please feel free to ask!
Last week, we
published the Exploring WebXPRT 4 white paper.
The paper describes the design and structure of WebXPRT 4, including detailed
information about the benchmark’s harness, HTML5 and WebAssembly capability
checks, and the structure of the performance test workloads. This week, to
help WebXPRT 4 testers understand how the benchmark calculates results, we’ve published
the WebXPRT 4 results calculation and confidence interval white
paper explains the WebXPRT 4 confidence interval and how it differs from typical
benchmark variability, and the formulas the benchmark uses to calculate the
individual workload scenario scores and overall score. The paper also provides
an overview of the statistical techniques WebXPRT uses to translate raw timings
the white paper’s discussion of the results calculation process, we’ve also
published a results calculation spreadsheet that shows the
raw data from a sample test run and reproduces the calculations WebXPRT uses to
produce workload scores and the overall score.
The paper is available on WebXPRT.com and on our XPRT white papers page. If you have any questions about the WebXPRT results calculation process, please let us know!