Recently, we informed XPRT blog readers that after updating Apple iPhones and iPads
to iOS and iPadOS 17, respectively, we began to see WebXPRT 4 failures on those
devices. In the Safari and Google Chrome browsers, WebXPRT 4 test runs were
freezing while running the Encrypt Notes and OCR Scan workload. We were able to
replicate the issue on every iOS/iPadOS 17 device we tested, and we also confirmed
that WebXPRT 4 continues to run without issues on other non-iOS platforms.
Our team has been investigating the situation, and we’ve made some progress. It’s clear that the failed test runs are getting stuck when the WASM-based Tesseract.js Optical Character Recognition (OCR) engine attempts to scan a shopping receipt. During our research, we’ve discovered an issue when the current Tesseract.js engine runs on iOS 17. This issue is broader than WebXPRT 4, and the Tesseract team is aware of the problem. Future versions of iOS 17 or later versions of Tesseract.js may include fixes for the problem, but unfortunately, we don’t know whether or when a fix will be available.
investigating possible workarounds for the problem, and hope to be able to
start testing soon. Our goal is that any solution we implement will not
significantly affect existing WebXPRT 4 scores on non-iOS 17 platforms.
We will continue to share any substantive progress updates with readers here in the blog. Once again, we apologize for any inconvenience this issue causes for WebXPRT 4 users, and we appreciate your patience while we work toward a solution. If you have any questions or comments, please feel free to contact us!
Yesterday, Apple revealed
the iPhone 15 and iPhone 15 Pro at its annual fall event, along with a new version of the iOS mobile operating system (iOS 17). The official iOS
17 launch will take place on September 18th, but before then, users
of newer iPhones can install the OS via the Apple Beta Software Program.
Today, a tech journalist informed us that during their testing of iPhone 15 and iPhone 15 Pro with iOS 17 Beta models, WebXPRT 4 has been freezing while running the Encrypt Notes and OCR Scan workload in the Safari 17 browser. Here in the lab, we were able to immediately replicate the issue on an iPhone 12 Pro with iOS 17 Beta model.
troubleshooting confirmed that WebXPRT 3 successfully runs to completion on iOS
17 Beta, so it appears that the problem is specific to WebXPRT 4. We also
confirmed that WebXPRT 4 freezes at the same place when running in the Google
Chrome browser on iOS 17 Beta, so we know that the problem does not occur only in
investigating the issue, and will publish our findings here in the blog as soon
as we feel confident that we’ve identified both the root cause and a workable
solution, if a solution is necessary. One reason a solution would not be
necessary is that the issue is a bug on the iOS 17 Beta side that Apple will resolve
before the official launch.
We apologize for any inconvenience this issue might cause for tech reviewers and iPhone users, and we appreciate your patience while we figure out what’s going on. If you have any questions about WebXPRT 4, please don’t hesitate to ask!
Recent visitors to CrXPRT.com may have seen a notice that encourages visitors to use WebXPRT 4
instead of CrXPRT 2 for performance testing on high-end Chromebooks. The notice
reads as follows:
Chromebook technology has progressed rapidly since we released CrXPRT 2, and
we’ve received reports that some CrXPRT 2 workloads may not stress top-bin
Chromebook processors enough to give the necessary accuracy for users to
compare their performance. So, for the latest test to compare the performance
of high-end Chromebooks, we recommend using WebXPRT 4.
made this recommendation because of the evident limitations of the CrXPRT 2
performance workloads when testing newer high-end hardware. CrXPRT 2 itself is
not that old (2020), but when we created the CrXPRT 2 performance workloads, we
started with a core framework of CrXPRT 2015 performance workloads. In a
similar way, we built the CrXPRT 2015 workloads on a foundation of WebXPRT 2015
workloads. At the time, the harness and workload structures we used to ensure
WebXPRT 2015’s cross-browser capabilities provided an excellent foundation that
we could adapt for our new ChromeOS benchmark. Consequently, CrXPRT 2 is a close
developmental descendant of WebXPRT 2015. Some of the legacy WebXPRT
2015/CrXPRT 2 workloads do not stress current high-end processors—a limitation that
prevents effective performance testing differentiation—nor do they engage the
latest web technologies.
the past, the Chromebook market skewed heavily toward low-cost devices with down-bin,
inexpensive processors, making this limitation less of an issue. Now, however,
more Chromebooks offer top-bin processors on par with traditional laptops and
workstations. Because of the limitations of the CrXPRT 2 workloads, we now recommend
WebXPRT 4 for both cross-browser and ChromeOS performance testing on the latest
tools and libraries, modern WebAssembly workloads, and additional Web Workers
tasks that cover a wide range of performance requirements.
CrXPRT 2 continues to function as a capable performance and battery life
comparison test for many ChromeOS devices, WebXPRT 4 is a more appropriate tool
to use with new high-end devices. If you haven’t yet used WebXPRT 4 for
Chromebook comparison testing, we encourage you to give it a try!
If you have any questions or concerns about CrXPRT 2 or WebXPRT 4, please don’t hesitate to ask!
When we’ve released a
new version of an XPRT benchmark app, it’s been our practice for many years to
maintain a link to the previous version on the benchmark’s main page. For
example, visitors can start on the WebXPRT 4 homepage at WebXPRT.com and follow links to access
WebXPRT 3, WebXPRT 2015, and WebXPRT 2013. Historically, we’ve maintained these
links because labs and tech reviewers usually take a while to introduce a new
benchmark to their testing suite. Continued access to the older benchmarks also
allows users to quickly compare new devices to old devices without retesting
That being said, several of the XPRT pages currently contain links to benchmarks that we no longer actively support. Some of those benchmarks still function correctly, and testers occasionally use them, but a few no longer work on the latest versions of the operating systems or browsers that we designed them to test. While we want to continue to provide a way for longtime XPRT users to access legacy XPRTs, we also want to avoid potential confusion for new users. We believe our best way forward is to archive older tests in a separate part of the site.
In the coming weeks, we’ll
be moving several legacy XPRT benchmarks to an archive section of the site. Once
the new section is ready, we’ll link to it from the Extras drop-down menu at
the top of BenchmarkXPRT.com. The benchmarks will still be available in the
archive, but we will not actively support them or directly link to them from
the homepages of active XPRTs.
During this process,
we’ll move the following benchmarks to the archive section:
- WebXPRT 2015 and 2013
- CrXPRT 2015
- HDXPRT 2014
- TouchXPRT 2014
- MobileXPRT 2015 and 2013
If you have any questions or concerns about the archive process or access to legacy XPRTs, please let us know!
Students of all ages will be starting a new school year over the next few weeks, and many learners will be shopping for tech devices that can help them excel in their studies. The tech marketplace can be confusing, and competing claims can be hard to navigate. The XPRTs are here to help! Whether you’re shopping for a new laptop, desktop, Chromebook, tablet, or phone, the XPRTs can provide reliable, industry-trusted performance scores that can cut through all the noise.
A good place to start looking
for scores is the WebXPRT 4 results viewer. The viewer displays WebXPRT 4 scores from almost
500 devices—including many hot new releases—and we’re adding new scores all the
time. To learn more about the viewer’s capabilities and how you can use it to
compare devices, check out this blog post.
Another resource we
offer is the XPRT results browser. The browser is the most efficient way to access the XPRT
results database, which currently holds more than 3,400 test results from over
140 sources, including major tech review publications around the world, OEMs,
and independent testers. It offers a wealth of current and historical
performance data across all of the XPRT benchmarks and hundreds of devices. You
can read more about how to use the results browser here.
Also, if you’re considering
a popular device, chances are good that a recent tech review includes an XPRT
score for that device. Two quick ways to find these reviews: (1) go to your
favorite tech review site and search for “XPRT” and (2) go to a search engine
and enter the device name and XPRT name (e.g. “Lenovo ThinkPad” and “WebXPRT”).
Here are a few recent tech reviews that use one of the XPRTs to evaluate a
- Notebookcheck used WebXPRT in reviews of the Acer Predator Helios 16, Samsung Galaxy A14 LTE, Apple MacBook Air 15 (M2, 2023), Google Pixel 7a, and in a recent article titled, “The Best Smartphones.”
- Gadgets360 used WebXPRT in reviews of the Apple Mac Mini (M2 Pro, 2023) and Apple MacBook Air 15 (M2, 2023).
- PCMag used CrXPRT 2 to review the Lenovo Flex 3 Chromebook, Acer Chromebook 315, and Acer Chromebook Vero 514.
- PCWorld used CrXPRT 2 in features called, “Best Chromebooks 2023: Best overall, best battery life, and more” and “The best laptops for kids: Best overall, best battery life, and more.”
- Tom’s Guide used HDXPRT 4 in reviews of the Alienware Aurora R15 and Dell XPS 8960 desktops.
The XPRTs can help back-to-school shoppers make better-informed and more confident tech purchases. As this school year begins, we hope you’ll find the data you need on our site or in an XPRT-related tech review. If you have any questions about the XPRTs, XPRT scores, or the results database please feel free to ask!
As part of our commitment to publishing reliable, unbiased benchmarks, we strive to make the XPRT development process as transparent as possible. In the technology assessment industry, it’s not unusual for people to claim that any given benchmark contains hidden biases, so we take preemptive steps to address this issue by publishing XPRT benchmark source code, detailed system disclosures and test methodologies, and in-depth white papers. Today, we’re focusing on the XPRT white paper library.
XPRT white paper library currently contains 21 white papers that we’ve
published over the last 12 years. We started publishing white papers to provide
XPRT users with more information about how we design our benchmarks, why we
make certain development decisions, and how the benchmarks work. If you have
questions about any aspect of one of the XPRT benchmarks, the white paper
library is a great place to find some answers.
example, the Exploring WebXPRT 4 white
paper describes the design and structure of WebXPRT 4, including detailed
information about the benchmark’s harness, HTML5 and WebAssembly (WASM)
capability checks, and the structure of the performance test workloads. It also
includes explanations of the benchmark’s scoring methodology, how to automate
tests, and how to submit results for publication.
companion WebXPRT 4 results calculation white paper explains the formulas that
WebXPRT 4 uses to calculate the individual workload scenario scores and overall
score, provides an overview of the statistical techniques WebXPRT uses to
translate raw timings into scores, and explains the benchmark’s confidence
interval and how it differs from typical benchmark variability. To supplement
the white paper’s discussion of the results calculation process, we published
a results calculation spreadsheet that shows the raw data from a sample test run and
reproduces the exact calculations WebXPRT uses to produce test scores.
hope that the XPRT white paper library will prove to be a useful resource for
you. If you have questions about any of our white papers, or suggestions for
topics that you’d like us to cover in possible future white papers, please let us know!