Over the past few months, we’ve been excited to see a substantial increase in the total number of completed WebXPRT runs. To put the increase in perspective, we had more total WebXPRT runs last month alone (40,453) than we had in the first two years WebXPRT was available (36,674)! This boost has helped us to reach two important milestones as we close in on the end of 2023.
The first milestone is that the number of WebXPRT 4 runs per month now exceeds the number of WebXPRT 3 runs per month. When we release a new version of an XPRT benchmark, it can take a while for users to transition from using the older version. For OEM labs and tech journalists, adding a new benchmark to their testing suite often involves a significant investment in back testing and gathering enough test data for meaningful comparisons. When the older version of the benchmark has been very successful, adoption of the new version can take longer. WebXPRT 3 has been remarkably popular around the world, so we’re excited to see WebXPRT 4 gain traction and take the lead even as the total number of WebXPRT runs increases each month. The chart below shows the number of WebXPRT runs per month for each version of WebXPRT over the past ten years. WebXPRT 4 usage first surpassed WebXPRT 3 in August of this year, and after looking at data for the last three months, we think its lead is here to stay.
The second important milestone is the cumulative number of WebXPRT runs, which recently passed 1.25 million, as the chart below shows. For us, this moment represents more than a numerical milestone. For a benchmark to succeed, developers need the trust and support of the benchmarking community. WebXPRT’s consistent year-over-year growth tells us that the benchmark continues to hold value for manufacturers, OEM labs, the tech press, and end users. We see it as a sign of trust that folks repeatedly return to the benchmark for reliable performance metrics. We’re grateful for that trust, and for everyone that has contributed to the WebXPRT development process over the years.
We look forward to seeing how far
WebXPRT’s reach can extend in 2024! If you have any
questions or comments about using WebXPRT, let us know!
Over the past several weeks, we’ve
been working to find a solution to a problem with WebXPRT 4
test failures on Apple devices running iOS 17/17.1, iPadOS 17/17.1, and macOS
Sonoma with Safari 17/17.1. While we put significant effort
into an updated WebXPRT version that would mitigate this issue, we are happy to
report that it now looks like we’ll be able to stick with the current version!
Last Thursday, Apple released the
iOS 17.2 beta for participants in the Apple Developer Program. When we tested
the current version of WebXPRT 4 on iOS 17.2, the tests completed without any
issues. We then successfully completed tests on iPadOS 17.2 and macOS Sonoma
14.2 with Safari 17.2. Now that we have good reasons to believe that the iOS
17.2 release will solve the problem, sticking with the current WebXPRT 4 build
will maximize continuity and minimize disruption for WebXPRT users.
Apple has not yet published a public
release date for iOS/iPad OS/Safari 17.2. Based on past development schedules,
it seems likely that they will release it between mid-November and early
December, but that’s simply our best guess. Until then, users who want to test
WebXPRT 4 on devices running iOS 17/17.1, iPadOS 17/17.1, or macOS Sonoma with
Safari 17/17.1 will need to update those devices to iOS/iPad OS/Safari 17.2 via
the Apple Developer Program.
To help Apple users better navigate
testing until the public 17.2 release, we’ve added a function to the current
WebXPRT 4 start page that will notify users if they need to update their
operating system to test.
We appreciate everyone’s patience as
we worked to find a solution to this problem! If you have any questions or
concerns about WebXPRT 4, please let us know.
In recent blog posts, we discussed
an issue that we encountered when attempting to run WebXPRT 4 on iOS 17 devices.
If you missed those posts, you can find more details about the nature of the
problem here. In
short, the issue is that the Encrypt Notes and OCR scan subtest in WebXPRT 4
gets stuck when the Tesseract.js Optical Character Recognition (OCR) engine
attempts to scan a shopping receipt. We’ve verified that the issue occurs on
devices running iOS 17, iPadOS 17, and macOS Sonoma with Safari 17.
After a good bit of troubleshooting and research to try and identify the cause of the problem, we decided to build an updated version of WebXPRT 4 that uses a newer version of Tesseract for the OCR task. Aside from updating Tesseract in the new build, we aimed to change as little as possible. To try and maximize continuity, we’re still using the original input image for the receipt scanning task, and we decided to stick with using the WASM library instead of a WASM-SIMD library. Aside from a new version of tesseract.js, WebXPRT 4 version number updates, and updated documentation where necessary, all other aspects of WebXPRT 4 will remain the same.
testing a candidate build of this new version on a wide array of devices. The
results so far seem promising, but we want to complete our due diligence and
make sure this is the best approach to solving the problem. We know that OEM
labs and tech reviewers put a lot of time and effort into compiling databases
of results, so we hope to provide a solution that minimizes results disruption
and inconvenience for WebXPRT 4 users. Ideally, folks would be able to
integrate scores from the new build without any questions or confusion about comparability.
We don’t yet have an exact release date for a new WebXPRT 4 build, but we can say that we’re shooting for the end of October. We appreciate everyone’s patience as we work towards the best possible solution. If you have any questions or concerns about an updated version of WebXPRT 4, please let us know.
Recently, we informed XPRT blog readers that after updating Apple iPhones and iPads
to iOS and iPadOS 17, respectively, we began to see WebXPRT 4 failures on those
devices. In the Safari and Google Chrome browsers, WebXPRT 4 test runs were
freezing while running the Encrypt Notes and OCR Scan workload. We were able to
replicate the issue on every iOS/iPadOS 17 device we tested, and we also confirmed
that WebXPRT 4 continues to run without issues on other non-iOS platforms.
Our team has been investigating the situation, and we’ve made some progress. It’s clear that the failed test runs are getting stuck when the WASM-based Tesseract.js Optical Character Recognition (OCR) engine attempts to scan a shopping receipt. During our research, we’ve discovered an issue when the current Tesseract.js engine runs on iOS 17. This issue is broader than WebXPRT 4, and the Tesseract team is aware of the problem. Future versions of iOS 17 or later versions of Tesseract.js may include fixes for the problem, but unfortunately, we don’t know whether or when a fix will be available.
investigating possible workarounds for the problem, and hope to be able to
start testing soon. Our goal is that any solution we implement will not
significantly affect existing WebXPRT 4 scores on non-iOS 17 platforms.
We will continue to share any substantive progress updates with readers here in the blog. Once again, we apologize for any inconvenience this issue causes for WebXPRT 4 users, and we appreciate your patience while we work toward a solution. If you have any questions or comments, please feel free to contact us!
Yesterday, Apple revealed
the iPhone 15 and iPhone 15 Pro at its annual fall event, along with a new version of the iOS mobile operating system (iOS 17). The official iOS
17 launch will take place on September 18th, but before then, users
of newer iPhones can install the OS via the Apple Beta Software Program.
Today, a tech journalist informed us that during their testing of iPhone 15 and iPhone 15 Pro with iOS 17 Beta models, WebXPRT 4 has been freezing while running the Encrypt Notes and OCR Scan workload in the Safari 17 browser. Here in the lab, we were able to immediately replicate the issue on an iPhone 12 Pro with iOS 17 Beta model.
troubleshooting confirmed that WebXPRT 3 successfully runs to completion on iOS
17 Beta, so it appears that the problem is specific to WebXPRT 4. We also
confirmed that WebXPRT 4 freezes at the same place when running in the Google
Chrome browser on iOS 17 Beta, so we know that the problem does not occur only in
investigating the issue, and will publish our findings here in the blog as soon
as we feel confident that we’ve identified both the root cause and a workable
solution, if a solution is necessary. One reason a solution would not be
necessary is that the issue is a bug on the iOS 17 Beta side that Apple will resolve
before the official launch.
We apologize for any inconvenience this issue might cause for tech reviewers and iPhone users, and we appreciate your patience while we figure out what’s going on. If you have any questions about WebXPRT 4, please don’t hesitate to ask!
Recent visitors to CrXPRT.com may have seen a notice that encourages visitors to use WebXPRT 4
instead of CrXPRT 2 for performance testing on high-end Chromebooks. The notice
reads as follows:
Chromebook technology has progressed rapidly since we released CrXPRT 2, and
we’ve received reports that some CrXPRT 2 workloads may not stress top-bin
Chromebook processors enough to give the necessary accuracy for users to
compare their performance. So, for the latest test to compare the performance
of high-end Chromebooks, we recommend using WebXPRT 4.
made this recommendation because of the evident limitations of the CrXPRT 2
performance workloads when testing newer high-end hardware. CrXPRT 2 itself is
not that old (2020), but when we created the CrXPRT 2 performance workloads, we
started with a core framework of CrXPRT 2015 performance workloads. In a
similar way, we built the CrXPRT 2015 workloads on a foundation of WebXPRT 2015
workloads. At the time, the harness and workload structures we used to ensure
WebXPRT 2015’s cross-browser capabilities provided an excellent foundation that
we could adapt for our new ChromeOS benchmark. Consequently, CrXPRT 2 is a close
developmental descendant of WebXPRT 2015. Some of the legacy WebXPRT
2015/CrXPRT 2 workloads do not stress current high-end processors—a limitation that
prevents effective performance testing differentiation—nor do they engage the
latest web technologies.
the past, the Chromebook market skewed heavily toward low-cost devices with down-bin,
inexpensive processors, making this limitation less of an issue. Now, however,
more Chromebooks offer top-bin processors on par with traditional laptops and
workstations. Because of the limitations of the CrXPRT 2 workloads, we now recommend
WebXPRT 4 for both cross-browser and ChromeOS performance testing on the latest
tools and libraries, modern WebAssembly workloads, and additional Web Workers
tasks that cover a wide range of performance requirements.
CrXPRT 2 continues to function as a capable performance and battery life
comparison test for many ChromeOS devices, WebXPRT 4 is a more appropriate tool
to use with new high-end devices. If you haven’t yet used WebXPRT 4 for
Chromebook comparison testing, we encourage you to give it a try!