Back in January, we
discussed the ChromeOS team’s decision to eventually end
support for all user-installed Chrome Apps—including CrXPRT
2—upon
the release of Chrome 138 in July of this year. As best we can tell, the move
is part of their overall strategy of transitioning all support to Chrome
extensions and Progressive Web Apps. We knew that after
the support cutoff date, we would not be able to publish any fixes or updates
for CrXPRT 2, but we weren’t exactly sure how the transition would
affect the app’s overall functionality.
We’ve now confirmed that while CrXPRT 2 still functions normally through Chrome 138.0.7204.255 (beta), the app does not launch at all on Chrome Canary 139. Consequently, we expect that stable channel system updates will disable CrXPRT 2 on most systems after Chrome 139 goes live on August 5th. We will initially leave CrXPRT 2 on our site for those who want to use it on older versions of Chrome, but over time we will archive it as an inactive benchmark.
We want to extend our heartfelt thanks to the many
people around the world who used CrXPRT 2 for lab evaluations, product reviews,
and individual testing over the past several years. We’re grateful for your
support! We will update readers here in the blog if we decide to pursue new
ChromeOS benchmark development work in the future.
One of the strengths of WebXPRT is that it’s a remarkably easy benchmark to run. Its upfront simplicity attracts users with a wide range of technical skills—everyone from engineers in cutting-edge OEM labs to veteran tech journalists to everyday folks who simply want to test their gear’s browser performance. With so many different kinds of people running the test each day, it’s certain that at least some of them use very different approaches to testing. In today’s blog, we’re going to share some of the key benchmarking practices we follow in the XPRT lab—and encourage you to consider—in order to produce the most consistent and reliable WebXPRT scores.
We offer
these best practices as tips you might find useful in your testing. Each step relates
to evaluating browser performance with WebXPRT, but several of these practices will
apply to other benchmarks as well.
Test with clean images: In the XPRT lab, we typically use an out-of-box (OOB) method for testing new devices. OOB testing means that other than running the initial OS and browser version updates that users are likely to run after first turning on the device, we change as little as possible before testing. We want to assess the performance that buyers are likely to see when they first purchase the device and before they install additional software. This approach is the best way to provide an accurate assessment of the performance retail buyers will experience from their new devices. That said, the OOB method is not appropriate for certain types of testing, such as when you want to compare largely identical systems or when you want to remove as much pre-loaded software as possible. The OOB method is less relevant to users who want to see how their device performs as it is.
Browser updates can have a significant impact: Most people know that different browsers often produce different performance scores on the same system. They may not know that there can be shifts in performance between different versions of the same browser. While most browser updates don’t have a large impact on performance, a few updates have increased (or even decreased) browser performance by a significant amount. For this reason, it’s always important to record and disclose the extended browser version number for each test run. The same principle applies to any other relevant software.
Turn off automatic updates: We do our best to eliminate or minimize app and system updates after initial setup. Some vendors are making it more difficult to turn off updates completely, but you should always double-check update settings before testing. On Windows systems, the same considerations apply to turning off User Account Control notifications.
Let the system settle: Depending on the system and the OS, a significant amount of system-level activity can be going on in the background after you turn it on. As much as possible, we like to wait for a stable baseline (idle time) of system activity before kicking off a test. If we start testing immediately after booting the system, we often see higher variance in the first run before the scores start to tighten up.
Run the test more than once: Because of natural variance, our standard practice in the XPRT lab is to publish a score that represents the median of three to five runs, if not more. If you run a benchmark only once and the score differs significantly from other published scores, your result could be an outlier that you would not see again under stable testing conditions or over the course of multiple runs.
Clear the cache: Browser caching can improve web page performance, including the loading of the types of JavaScript and HTML5 assets that WebXPRT uses in its workloads. Depending on the platform under test, browser caching may or may not significantly change WebXPRT scores, but clearing the cache before testing and between each run can help improve the accuracy and consistency of scores.
We hope
these tips will serve as a good baseline methodology for your WebXPRT testing.
If you have any questions about WebXPRT, the other XPRTs, or benchmarking in
general, please let us know!
Here at the XPRTs,
our primary goal is to provide free, easy-to-use benchmark tools that can help
everyone—from OEM labs to tech press journalists to individual consumers—understand
how well devices will perform while completing everyday computing tasks. We
track progress toward that goal in several ways, but one of the most important
is how much people use and discuss the XPRTs. When the name of one of our apps
appears in an ad, article, or tech review, we call it a “mention.” Tracking
mentions helps us gauge our reach.
We
occasionally like to share a sample of recent XPRT mentions here in the blog.
If you just started following the XPRTs, it may be surprising to see our
program’s global reach. If you’re a longtime reader and you’re used to seeing
WebXPRT or CrXPRT in major tech press articles, it may be surprising to learn
more about overseas tech press publications or see how some government agencies
use the XPRTs to make decisions. In any case, we hope you’ll enjoy exploring
the links below!
Recent
mentions include:
Computerworld noted that the Polish government’s Ministry of Digital Affairs used WebXPRT to establish a minimum performance baseline for Chromebooks that could be eligible for their Laptops for Teachers program.
Other outlets that have published articles, ads, or reviews mentioning the XPRTs in the last few months include the following: 3DNews.ru (Russia), Acer, Alza.cz (Czech Republic), Android Headlines, Android.com.pl (Poland), BenchLife.info, ComputerBase (Germany), Dell, DGL.ru (Russia), eTeknix, Gadgety (Israel), GeekWeek (Poland), GSMArena.com, ID.nl (Netherlands), Intel, ITC.ua (Ukraine), ITMedia (Japan), Komputronik (Poland), Mashable, MSN, PC Games Hardware (Germany), PCMag, PurePC.pl (Poland), QQ.com (China), SlashGear, Sohu.com (China), TechHut, TechRadar, TechToday (Ukraine), Tom’s Hardware, Tool Elvaliant (Italy), Tweakers, and ZDNet, among others.
If you’d like to receive monthly updates on XPRT-related news and activity, we encourage you to sign up for the BenchmarkXPRT Development Community newsletter. It’s completely free, and all you need to do to join the newsletter mailing list is let us know! We won’t publish, share, or sell any of the contact information you provide, and we’ll only send you the monthly newsletter and occasional benchmark-related announcements, such as important news about patches or releases.
If
you have any questions about the XPRTs, suggestions, or requests for future
blog topics, please feel free to contact us.
Some of our readers have been
following the XPRTs since the early days, and they may remember using legacy
versions of benchmarks such as HDXPRT 2014 or WebXPRT 2013. For many years, whenever
we released a new version of a benchmark, we would maintain a link to the
previous version on the benchmark’s main page. However, as interest in the
older versions understandably waned and we stopped formally supporting them, many
of those legacy XPRTs stopped working on the latest versions of the operating
systems or browsers that we designed them to test. While we wanted to continue
to provide a way for users to access those legacy XPRTs, we also wanted to
avoid potential confusion for new users who might see links to old versions on
our site. We decided that the best solution was to archive older tests in a
separate section of the site—the XPRT archive.
Recently, as we discussed XPRT plans
for 2025, it became clear that we needed to add AIXPRT
and CloudXPRT
to the archive. Both benchmarks represent landmark efforts toward our ongoing
goal of providing cutting-edge performance assessment tools, but even though a
few tech press publications and OEM labs experimented with them, neither
benchmark gained enough widespread adoption to justify their continued support.
As a result, we decided to focus our resources elsewhere and halt development
on both benchmarks. Since then, ongoing updates to their respective software
components and target platforms have rendered them largely unusable. By
archiving both benchmarks, we hope to avoid any future confusion for visitors
who may otherwise try to use them.
Over the coming weeks, we’ll be
moving the AIXPRT and CloudXPRT installation packages to the XPRT archive page.
We’re grateful to everyone who has used AIXPRT and CloudXPRT in the past, and
we apologize for any inconvenience this change may cause.
If you have any questions or concerns about access to either of these benchmarks—or about anything else related to the XPRTs, please let us know!
Once or twice per year, we
refresh our ongoing series of WebXPRT comparison
tests to see if software version updates have reordered the performance
rankings of popular web browsers. We published our most recent comparison last
June, when we used WebXPRT 4 to compare the performance of five browsers—Brave,
Google Chrome, Microsoft Edge, Mozilla Firefox, and Opera—on a Lenovo ThinkPad
T14s Gen 3. When assessing performance differences, it’s worth noting that all
the browsers—except for Firefox—are built on a Chromium foundation. In the last
round of tests, the scores were very tight, with a difference of only four
percent between the last-place browser (Brave) and the winner (Chrome). Firefox’s
score landed squarely in the middle of the pack.
Recently, we conducted a new set
of tests to see how performance scores may have changed. To maintain continuity
with our last comparison, we stuck with the same ThinkPad T14s as our reference
system. That laptop is still in line with current mid-range laptops, so our
comparison scores are likely to fall within the range of scores we would see
from a typical user today. The ThinkPad is equipped with an Intel Core i7-1270P
processor and 16 GB of RAM, and it’s running Windows 11 Pro, version 23H2
(22631.4890).
Before testing, we installed all
current Windows updates, and we updated each of the browsers to the latest
available stable version. After the update process was complete, we turned off
updates to prevent any interference with test runs. We ran WebXPRT 4 five times
on each of the five browsers. In Figure 1 below, each browser’s score is the
median of the five test runs.
In this round of tests, the gap widened a bit between first and last place scores, with a difference of just over six percent between the lowest median score of 303 (Brave) and the highest median score of 322 (Firefox).
Figure 1: The median scores from running WebXPRT 4 five times with each browser on the Lenovo ThinkPad T14s Gen 3.
In this round of tests, the
distribution of scores indicates that most users would not see a significant
performance difference if they switched between the latest versions of these
browsers. The one exception may be a change from the latest version of Brave to
the latest version of Firefox. Even then, the quality of your browsing
experience will often depend on other factors. The types of things you do on
the web (e.g., gaming, media consumption, or multi-tab browsing), the type and
number of extensions you’ve installed, and how frequently the browsers issue
updates and integrate new technologies—among other things—can all affect
browser performance over time. It’s important to keep such variables in mind
when thinking about how browser performance comparison results may translate to
your everyday web experience.
Have you tried using WebXPRT 4 in
your own browser performance comparison? If so, we’d love to hear about it!
Also, please let us know if
there are other types of WebXPRT comparisons you’d like to see!
CrXPRT users may remember that back in 2022, we discussed the ChromeOS team’s
decision to end formal support for Chrome Apps and instead
focus on Chrome extensions and Progressive Web Apps. This decision meant that
we would not be able to publish any future fixes or updates for CrXPRT 2,
although moving forward, we weren’t sure how it would affect the app’s
functionality.
After
receiving a lot of feedback regarding their original timeline, the ChromeOS
team decided to
extend Chrome App support for Enterprise and Education account customers
through January 2025. Because we publish CrXPRT through a private BenchmarkXPRT
developer account, we assumed at the time that the support extension would not apply
to CrXPRT.
Recently,
the ChromeOS team released new information about their scheduled support
timeline. Now, they plan to end formal support for all user-installed Chrome
Apps in July 2025 (Chrome 138). In February 2028, the Chrome 168 release will
mark the end of life for all Chrome Apps.
The good news is that—in spite of a lack of formal ChromeOS support over the past couple of years—the CrXPRT 2 performance and battery life tests have continued to run without any known issues. As of today, the app functions normally up through the Beta release of ChromeOS version 132.0.6834.52.
We will continue to
run the benchmark on a regular basis to monitor functionality, and we will
disclose any future issues here in the blog and on CrXPRT.com. We hope the app
will continue to run both performance and battery life tests well into the
future. However, given the frequency of Chrome updates, it’s difficult for us
to predict how long the benchmark will remain viable.
If you have any
questions about CrXPRT, please let us know!
Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.
