Recent visitors to CrXPRT.com may have seen a notice that encourages visitors to use WebXPRT 4
instead of CrXPRT 2 for performance testing on high-end Chromebooks. The notice
reads as follows:
Chromebook technology has progressed rapidly since we released CrXPRT 2, and
we’ve received reports that some CrXPRT 2 workloads may not stress top-bin
Chromebook processors enough to give the necessary accuracy for users to
compare their performance. So, for the latest test to compare the performance
of high-end Chromebooks, we recommend using WebXPRT 4.
made this recommendation because of the evident limitations of the CrXPRT 2
performance workloads when testing newer high-end hardware. CrXPRT 2 itself is
not that old (2020), but when we created the CrXPRT 2 performance workloads, we
started with a core framework of CrXPRT 2015 performance workloads. In a
similar way, we built the CrXPRT 2015 workloads on a foundation of WebXPRT 2015
workloads. At the time, the harness and workload structures we used to ensure
WebXPRT 2015’s cross-browser capabilities provided an excellent foundation that
we could adapt for our new ChromeOS benchmark. Consequently, CrXPRT 2 is a close
developmental descendant of WebXPRT 2015. Some of the legacy WebXPRT
2015/CrXPRT 2 workloads do not stress current high-end processors—a limitation that
prevents effective performance testing differentiation—nor do they engage the
latest web technologies.
the past, the Chromebook market skewed heavily toward low-cost devices with down-bin,
inexpensive processors, making this limitation less of an issue. Now, however,
more Chromebooks offer top-bin processors on par with traditional laptops and
workstations. Because of the limitations of the CrXPRT 2 workloads, we now recommend
WebXPRT 4 for both cross-browser and ChromeOS performance testing on the latest
tools and libraries, modern WebAssembly workloads, and additional Web Workers
tasks that cover a wide range of performance requirements.
CrXPRT 2 continues to function as a capable performance and battery life
comparison test for many ChromeOS devices, WebXPRT 4 is a more appropriate tool
to use with new high-end devices. If you haven’t yet used WebXPRT 4 for
Chromebook comparison testing, we encourage you to give it a try!
If you have any questions or concerns about CrXPRT 2 or WebXPRT 4, please don’t hesitate to ask!
Each month, we send a newsletter to members of the BenchmarkXPRT Development Community. In the newsletter, we recap the latest updates from the XPRT world and provide a summary of the previous month’s XPRT-related activity, including uses or mentions of the XPRTs in the tech press. More people read the weekly XPRT blog than receive the monthly newsletter, so we realized that some blog readers may be unaware of the wide variety of tech outlets that regularly use or mention the XPRTs.
So for today’s blog, we want to give readers a sampling of the XPRT press usage we see on a weekly basis. Recent mentions include:
- Tom’s Guide used HDXPRT 4 to compare the performance of the Geekom Mini IT8 and Dell OptiPlex 7090 Ultra small-form-factor PCs.
- Intel used WebXPRT 4 test data in promotional material for their line of 12th Gen) Intel Core processors(Alder Lake). Hundreds of press outlets then republished the presentation.
- AnandTech used WebXPRT 4 to evaluate the Cincoze DS-1300 Industrial PC.
- ZDNet used CrXPRT 2 in a review titled The best Chromebooks for students: Student-proof laptops.
- PCWorld used CrXPRT 2 to provide data for an article listing their top Chromebook recommendations.
- TechPowerUp used WebXPRT 3 to compare the browser performance of Intel Core i9-12900KS processor-based systems and other Intel- and AMD processor-based systems.
- Other outlets that have published articles, ads, or reviews mentioning the experts in the last few months include: Android Authority, ASUS, BenchLife, Gadgets 360, Good Gear Guide, Hardware.info, Hot Hardware, ITHardware (Poland), ITMedia (Japan), Itndaily (Russia), Mobile01.com (China), Notebookcheck, PCMag, ProClockers, Sohu.com (China), Tom’s Hardware, and Tweakers.
If you don’t currently receive the monthly
BenchmarkXPRT newsletter, but would like to join the mailing list, please let us know! We will not publish or sell any of the contact information you
provide, and will only send the monthly newsletter and occasional benchmark-related
announcements such as patch notifications or new benchmark releases.
Recently, a tester contacted us with details from a CrXPRT 2 performance test run that they’d successfully completed on… an Apple MacBook Pro! Because CrXPRT 2 is a Chrome Web App that we designed for Chrome OS, it was quite a surprise to hear that it is now possible to run CrXPRT 2 on non-Chrome OS platforms by using FydeOS.
FydeOS is an operating system based on a fork of the Chromium OS project. Developers originally intended FydeOS to be a Google-independent, Chrome-like alternative for the Chinese educational market, but FydeOS is now available to the English-speaking consumer and enterprise markets as well. FydeOS users can run a Chrome-like OS on something other than a Chromebook or a Chromebox, such as a PC, Mac, virtual machine, or even a Raspberry Pi device. Additionally, FydeOS supports Android, Chrome OS, and Linux apps, and users can run those apps at the same time on the same screen.
We have not yet conducted any testing with FydeOS in our lab, but we wanted to pass along this information to any readers who may be interested. If the OS operates as described, it may provide a way for us to experiment with using CrXPRT 2 in some interesting cross-platform tests.
Last March, we discussed the Chrome OS team’s original announcement that they would be phasing out support for Chrome Apps altogether in June 2021, and would shift their focus to Chrome extensions and Progressive Web Apps. The Chrome OS team eventually extended support for existing Chrome Apps through June 2022, but as of this week, we see no indication that they will further extend support for Chrome Apps published with general developer accounts. If the end-of-life schedule for Chrome Apps does not change in the next few months, both CrXPRT 2 and CrXPRT 2015 will stop working on new versions of Chrome OS at some point in June.
To maintain CrXPRT
functionality past June, we would need to rebuild the app completely—either as
a Progressive Web App or in some other form. For this reason, we want to
reassess our approach to Chrome OS testing, and investigate which features and
technologies to include in a new Chrome OS benchmark. Our current goal is to
gather feedback and conduct exploratory research over the next few months, and begin
developing an all-new Chrome OS benchmark for publication by the end of the
While we will discuss ideas for this new Chrome OS benchmark in future blog posts, we welcome ideas from CrXPRT users now. What features or workloads would you like the new benchmark to retain? Would you like us to remove any components from the existing benchmark? Does the battery life test in its current form suit your needs? If you have any thoughts about these questions or any other aspects of Chrome OS benchmarking, please let us know!
we heard from a BenchmarkXPRT Development Community member who was testing
Chromebooks in their lab. On a few of the Chromebooks, they saw sporadic CrXPRT
2 battery life test failures where CrXPRT 2 would successfully complete a
battery life test and produce a result for the initial run, but then fail at
the end of later runs.
a considerable amount of troubleshooting, they determined that the issue seemed
to be related to the way some systems automatically shut down before the
battery is completely exhausted, and the way some systems will automatically
boot up once the tester plugs in the power adapter for charging. This member found
that when they added a few system configuration steps before battery life tests
and made slight changes to their post-test routine, the systems that had
previously experienced consistent failures would successfully complete battery
life tests and produce results.
added steps are quick and straightforward, and we decided to add them to the
Configuring the test device and Running the tests sections of the CrXPRT 2 user manual.
We hope this updated guidance will help to prevent future frustration for
CrXPRT 2 testers.
If you have any questions or comments about the CrXPRT 2 battery life test, please feel free to contact us!
Recently, we had a discussion with a community member about why we no longer recommend specific screen brightness settings during CrXPRT 2 battery life tests. In the CrXPRT 2015 user manual, we recommended setting the test system’s screen brightness to 200 nits. Because the amount of power that a system directs to screen brightness can have a significant impact on battery life, we believed that pegging screen brightness to a common standard for all test systems would yield apple-to-apples comparisons.
After extensive experience with CrXPRT 2015 testing, we decided to not recommend a standard screen brightness with CrXPRT 2, for the following reasons:
- A significant number of Chromebooks cannot produce a screen brightness of 200 nits. A few higher-end models can do so, but they are not representative of most Chromebooks. Some Chromebooks, especially those that many school districts and corporations purchase in bulk, cannot produce a brightness of even 100 nits.
- Because of the point above, adjusting screen brightness would not represent real-life conditions for most Chromebooks, and the battery life results could mislead consumers who want to know the battery life they can expect with default out-of-box settings.
- Most testers, and even some labs, do not have light meters, and the simple brightness percentages that the operating system reports produce different degrees of brightness on different systems. For testers without light meters, a standardized screen brightness recommendation could discourage them from running the test.
- The brightness controls for some low-end Chromebooks lack the fine-tuning capability that is necessary to standardize brightness between systems. In those cases, an increase or decrease of one notch can swing brightness by 20 to 30 nits in either direction. This could also discourage testing by leading people to believe that they lack the capability to correctly run the test.
situations where testers want to compare battery life using standardized screen
brightness, we recommend using light meters to set the brightness levels as
closely as possible. If the brightness levels between systems vary by more than
few nits, and if the levels vary significantly from out-of-box settings, the
publication of any resulting battery life results should include a full
disclosure and explanation of test conditions.
the majority of testers without light meters, running the CrXPRT 2 battery life
test with default screen brightness settings on each system provides a reliable
and accurate estimate of the type of real-world, out-of-box battery life
consumers can expect.
If you have any questions or comments about the CrXPRT 2 battery life test, please feel free to contact us!