Back in January, we
discussed the ChromeOS team’s decision to eventually end
support for all user-installed Chrome Apps—including CrXPRT
2—upon
the release of Chrome 138 in July of this year. As best we can tell, the move
is part of their overall strategy of transitioning all support to Chrome
extensions and Progressive Web Apps. We knew that after
the support cutoff date, we would not be able to publish any fixes or updates
for CrXPRT 2, but we weren’t exactly sure how the transition would
affect the app’s overall functionality.
We’ve now confirmed that while CrXPRT 2 still functions normally through Chrome 138.0.7204.255 (beta), the app does not launch at all on Chrome Canary 139. Consequently, we expect that stable channel system updates will disable CrXPRT 2 on most systems after Chrome 139 goes live on August 5th. We will initially leave CrXPRT 2 on our site for those who want to use it on older versions of Chrome, but over time we will archive it as an inactive benchmark.
We want to extend our heartfelt thanks to the many
people around the world who used CrXPRT 2 for lab evaluations, product reviews,
and individual testing over the past several years. We’re grateful for your
support! We will update readers here in the blog if we decide to pursue new
ChromeOS benchmark development work in the future.
In
our previous blog post, we discussed the rapidly
expanding influence of AI-enhanced technologies in areas like everyday browser
activity—and the growing need for objective performance data that can help us
understand how well new consumer devices will handle AI tasks. We noted that WebXPRT 4 already includes timed AI tasks
in two of its workloads—the “Organize Album using AI” and “Encrypt Notes and
OCR Scan”—and we provided some technical details for the Organize Album
workload. In today’s post, we’ll focus on the Encrypt Notes workload.
The Encrypt Notes workload
includes two separate scenarios that reflect common web-based productivity app
tasks. The first scenario syncs a set of encrypted notes, and the second
scenario uses AI-based optical character recognition (OCR) to scan a receipt,
extract data, and then add that data to an expense report.
Here
are the details for each scenario:
The
encrypt notes scenario downloads a set of notes, encrypts that data,
temporarily stores it in the browser’s localStorage object (the
localStorageDB.js database layer), and then decrypts and renders it for
display. This scenario measures HTML5 Local Storage, JavaScript, AES encryption, and WebAssembly (Wasm) performance.
The
OCR scan scenario uses a Wasm-based version of Tesseract.js (tesseract-core.wasm.js v2.20) to
scan an expense receipt. Tesseract.js is a JavaScript port of the Tesseract OCR engine—a popular open-source C/C++
library that extracts text from images and PDFs. The scenario then adds the
receipt to an expense report. This scenario measures HTML5 Local Storage,
JavaScript, and Wasm performance.
We mention this test under the
AI umbrella in part because people sometimes use the term “OCR” to refer to a
spectrum of AI and non-AI technologies. In this case, though, the specifics
make this workload clearly have an AI component. The Wasm-based Tesseract library
that we use in WebXPRT 4 is based on a version of C/C++ (v4.x) that uses Long Short-Term Memory (LSTM). LSTM is a type of recurrent neural network (RNN) that is particularly
well-suited for processing and predicting sequential data. As such, it is
clearly an AI component of the Encrypt Notes and OCR Scan workload.
To
produce a score for each iteration of the workload, WebXPRT calculates the
total time that it takes for a system to sync (encrypt, decrypt, and render)
the notes, use OCR to scan the receipt, and add the scanned data to an expense
report. In a standard test, WebXPRT runs seven iterations of the entire
six-workload performance suite before calculating an overall test score. You
can find out more about the WebXPRT results calculation process here.
Along
with our post on the Organize Album workload, we hope this information
provides a deeper understanding of WebXPRT 4’s AI-equipped workloads. As we
mentioned last time, if you want to explore the structure of these workloads in
more detail, you can check out previous blog posts for information about how to
access and use the WebXPRT 4 source code for
free. You can also read more about WebXPRT’s overall structure and other
workloads in the Exploring WebXPRT 4 white paper.
If you have any questions about WebXPRT 4, please let us know!
I recently revisited an XPRT blog entry that we posted from CES Las
Vegas back in 2020. In that post, I reflected on the show’s expanded AI
emphasis, and I wondered if we were reaching a tipping point where AI-enhanced
and AI-driven tools and applications would become a significant presence in
people’s daily lives. It felt like we were approaching that point back then
with the prevalence of AI-powered features such as image enhancement and text
recommendation, among many others. Now, seamless AI integration with common
online tasks has become so widespread that many people unknowingly benefit from
AI interactions several times a day.
As
AI’s role in areas like everyday browser activity continues to grow—along with
our expectations for what our consumer devices should be able to
handle—reliable AI-oriented benchmarking is more vital than ever. We need
objective performance data that can help us understand how well a new desktop,
laptop, tablet, or phone will handle AI tasks.
WebXPRT 4 already includes timed AI tasks
in two of its workloads: the “Organize Album using AI” workload and
the “Encrypt Notes and OCR Scan” workload. These two workloads
reflect the types of light browser-side inference tasks that are now fairly
common in consumer-oriented web apps and extensions. In today’s post, we’ll
provide some technical information about the Organize Album workload. In a
future post, we’ll do the same for the Encrypt Notes workload.
The
Organize Album workload includes two different timed tasks that reflect a
common scenario of organizing online photo albums. The workload utilizes the AI
inference and JavaScript capabilities of the WebAssembly (Wasm) version of OpenCV.js—an
open-source computer vision and machine learning library. In WebXPRT 4, we used
OpenCV.js version 4.5.2.
Here are the details for each task:
The first task measures the time it takes to complete a face detection job with a set of five 720 x 480 photos that we sourced from commercial photo sites. The workload loads a Caffe deep learning framework model (res10_300x300_ssd_iter_140000_fp16.caffemodel) using the commands found here.
The second task measures the time it takes to complete an image classification job (labeling based on object detection) with a different set of five 718 x 480 photos that we sourced from the ImageNet computer vision dataset. The workload loads an ONNX-based SqueezeNet machine learning model (squeezenet.onnx v 1.0) using the commands found here.
To produce a score for each
iteration of the workload, WebXPRT calculates the total time that it takes for
a system to organize both albums. In a standard test, WebXPRT runs seven
iterations of the entire six-workload performance suite before calculating an
overall test score. You can find out more about the WebXPRT results calculation process here.
We
hope this post will give you a better sense of how WebXPRT 4 measures one kind
of AI performance. As a reminder, if you want to dig into the details at a more
granular level, you can access the WebXPRT 4 source code for free. In previous
blog posts, you can find information about how to access and use the code. You can also read
more about WebXPRT’s overall structure and other workloads in the Exploring WebXPRT 4 white paper.
If
you have any questions about this workload or any other aspect of WebXPRT 4,
please let us know!
One of the strengths of WebXPRT is that it’s a remarkably easy benchmark to run. Its upfront simplicity attracts users with a wide range of technical skills—everyone from engineers in cutting-edge OEM labs to veteran tech journalists to everyday folks who simply want to test their gear’s browser performance. With so many different kinds of people running the test each day, it’s certain that at least some of them use very different approaches to testing. In today’s blog, we’re going to share some of the key benchmarking practices we follow in the XPRT lab—and encourage you to consider—in order to produce the most consistent and reliable WebXPRT scores.
We offer
these best practices as tips you might find useful in your testing. Each step relates
to evaluating browser performance with WebXPRT, but several of these practices will
apply to other benchmarks as well.
Test with clean images: In the XPRT lab, we typically use an out-of-box (OOB) method for testing new devices. OOB testing means that other than running the initial OS and browser version updates that users are likely to run after first turning on the device, we change as little as possible before testing. We want to assess the performance that buyers are likely to see when they first purchase the device and before they install additional software. This approach is the best way to provide an accurate assessment of the performance retail buyers will experience from their new devices. That said, the OOB method is not appropriate for certain types of testing, such as when you want to compare largely identical systems or when you want to remove as much pre-loaded software as possible. The OOB method is less relevant to users who want to see how their device performs as it is.
Browser updates can have a significant impact: Most people know that different browsers often produce different performance scores on the same system. They may not know that there can be shifts in performance between different versions of the same browser. While most browser updates don’t have a large impact on performance, a few updates have increased (or even decreased) browser performance by a significant amount. For this reason, it’s always important to record and disclose the extended browser version number for each test run. The same principle applies to any other relevant software.
Turn off automatic updates: We do our best to eliminate or minimize app and system updates after initial setup. Some vendors are making it more difficult to turn off updates completely, but you should always double-check update settings before testing. On Windows systems, the same considerations apply to turning off User Account Control notifications.
Let the system settle: Depending on the system and the OS, a significant amount of system-level activity can be going on in the background after you turn it on. As much as possible, we like to wait for a stable baseline (idle time) of system activity before kicking off a test. If we start testing immediately after booting the system, we often see higher variance in the first run before the scores start to tighten up.
Run the test more than once: Because of natural variance, our standard practice in the XPRT lab is to publish a score that represents the median of three to five runs, if not more. If you run a benchmark only once and the score differs significantly from other published scores, your result could be an outlier that you would not see again under stable testing conditions or over the course of multiple runs.
Clear the cache: Browser caching can improve web page performance, including the loading of the types of JavaScript and HTML5 assets that WebXPRT uses in its workloads. Depending on the platform under test, browser caching may or may not significantly change WebXPRT scores, but clearing the cache before testing and between each run can help improve the accuracy and consistency of scores.
We hope
these tips will serve as a good baseline methodology for your WebXPRT testing.
If you have any questions about WebXPRT, the other XPRTs, or benchmarking in
general, please let us know!
Some of our readers have been
following the XPRTs since the early days, and they may remember using legacy
versions of benchmarks such as HDXPRT 2014 or WebXPRT 2013. For many years, whenever
we released a new version of a benchmark, we would maintain a link to the
previous version on the benchmark’s main page. However, as interest in the
older versions understandably waned and we stopped formally supporting them, many
of those legacy XPRTs stopped working on the latest versions of the operating
systems or browsers that we designed them to test. While we wanted to continue
to provide a way for users to access those legacy XPRTs, we also wanted to
avoid potential confusion for new users who might see links to old versions on
our site. We decided that the best solution was to archive older tests in a
separate section of the site—the XPRT archive.
Recently, as we discussed XPRT plans
for 2025, it became clear that we needed to add AIXPRT
and CloudXPRT
to the archive. Both benchmarks represent landmark efforts toward our ongoing
goal of providing cutting-edge performance assessment tools, but even though a
few tech press publications and OEM labs experimented with them, neither
benchmark gained enough widespread adoption to justify their continued support.
As a result, we decided to focus our resources elsewhere and halt development
on both benchmarks. Since then, ongoing updates to their respective software
components and target platforms have rendered them largely unusable. By
archiving both benchmarks, we hope to avoid any future confusion for visitors
who may otherwise try to use them.
Over the coming weeks, we’ll be
moving the AIXPRT and CloudXPRT installation packages to the XPRT archive page.
We’re grateful to everyone who has used AIXPRT and CloudXPRT in the past, and
we apologize for any inconvenience this change may cause.
If you have any questions or concerns about access to either of these benchmarks—or about anything else related to the XPRTs, please let us know!
Once or twice per year, we
refresh our ongoing series of WebXPRT comparison
tests to see if software version updates have reordered the performance
rankings of popular web browsers. We published our most recent comparison last
June, when we used WebXPRT 4 to compare the performance of five browsers—Brave,
Google Chrome, Microsoft Edge, Mozilla Firefox, and Opera—on a Lenovo ThinkPad
T14s Gen 3. When assessing performance differences, it’s worth noting that all
the browsers—except for Firefox—are built on a Chromium foundation. In the last
round of tests, the scores were very tight, with a difference of only four
percent between the last-place browser (Brave) and the winner (Chrome). Firefox’s
score landed squarely in the middle of the pack.
Recently, we conducted a new set
of tests to see how performance scores may have changed. To maintain continuity
with our last comparison, we stuck with the same ThinkPad T14s as our reference
system. That laptop is still in line with current mid-range laptops, so our
comparison scores are likely to fall within the range of scores we would see
from a typical user today. The ThinkPad is equipped with an Intel Core i7-1270P
processor and 16 GB of RAM, and it’s running Windows 11 Pro, version 23H2
(22631.4890).
Before testing, we installed all
current Windows updates, and we updated each of the browsers to the latest
available stable version. After the update process was complete, we turned off
updates to prevent any interference with test runs. We ran WebXPRT 4 five times
on each of the five browsers. In Figure 1 below, each browser’s score is the
median of the five test runs.
In this round of tests, the gap widened a bit between first and last place scores, with a difference of just over six percent between the lowest median score of 303 (Brave) and the highest median score of 322 (Firefox).
Figure 1: The median scores from running WebXPRT 4 five times with each browser on the Lenovo ThinkPad T14s Gen 3.
In this round of tests, the
distribution of scores indicates that most users would not see a significant
performance difference if they switched between the latest versions of these
browsers. The one exception may be a change from the latest version of Brave to
the latest version of Firefox. Even then, the quality of your browsing
experience will often depend on other factors. The types of things you do on
the web (e.g., gaming, media consumption, or multi-tab browsing), the type and
number of extensions you’ve installed, and how frequently the browsers issue
updates and integrate new technologies—among other things—can all affect
browser performance over time. It’s important to keep such variables in mind
when thinking about how browser performance comparison results may translate to
your everyday web experience.
Have you tried using WebXPRT 4 in
your own browser performance comparison? If so, we’d love to hear about it!
Also, please let us know if
there are other types of WebXPRT comparisons you’d like to see!
Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.
