Recently, we had a discussion with a community member about why we no longer recommend specific screen brightness settings during CrXPRT 2 battery life tests. In the CrXPRT 2015 user manual, we recommended setting the test system’s screen brightness to 200 nits. Because the amount of power that a system directs to screen brightness can have a significant impact on battery life, we believed that pegging screen brightness to a common standard for all test systems would yield apple-to-apples comparisons.
After extensive experience with CrXPRT 2015 testing, we decided to not recommend a standard screen brightness with CrXPRT 2, for the following reasons:
A significant number of Chromebooks cannot produce a screen brightness of 200 nits. A few higher-end models can do so, but they are not representative of most Chromebooks. Some Chromebooks, especially those that many school districts and corporations purchase in bulk, cannot produce a brightness of even 100 nits.
Because of the point above, adjusting screen brightness would not represent real-life conditions for most Chromebooks, and the battery life results could mislead consumers who want to know the battery life they can expect with default out-of-box settings.
Most testers, and even some labs, do not have light meters, and the simple brightness percentages that the operating system reports produce different degrees of brightness on different systems. For testers without light meters, a standardized screen brightness recommendation could discourage them from running the test.
The brightness controls for some low-end Chromebooks lack the fine-tuning capability that is necessary to standardize brightness between systems. In those cases, an increase or decrease of one notch can swing brightness by 20 to 30 nits in either direction. This could also discourage testing by leading people to believe that they lack the capability to correctly run the test.
situations where testers want to compare battery life using standardized screen
brightness, we recommend using light meters to set the brightness levels as
closely as possible. If the brightness levels between systems vary by more than
few nits, and if the levels vary significantly from out-of-box settings, the
publication of any resulting battery life results should include a full
disclosure and explanation of test conditions.
the majority of testers without light meters, running the CrXPRT 2 battery life
test with default screen brightness settings on each system provides a reliable
and accurate estimate of the type of real-world, out-of-box battery life
consumers can expect.
If you have any questions or comments about the CrXPRT 2 battery life test, please feel free to contact us!
Last month, we discussed a potential fix for the error that was preventing CrXPRT 2 testers from successfully completing battery life tests on systems running Chrome v89.x and later. Since then, we’ve been testing an updated, unpublished version of the app package across several Chromebook models to ensure that the new build is stable and produces consistent results. We’re happy to report that our testing was successful, and we’ve published the new CrXPRT build (v18.104.22.168) in the Chrome Web Store and it is live as of 12:45 PM EDT today.
that it might take some time for the update to appear on your Chromebook and,
once it does, you might have to manually approve the update notice.
the tests nor the method of calculating the overall score and battery-life
score in this new build have changed, so results are comparable with previous
CrXPRT 2 results.
We appreciate everyone’s patience while we found a solution to the error. If you have any questions or comments about the CrXPRT 2 battery life test, please feel free to contact us!
the past few months, we’ve been recommending that CrXPRT 2
testers not use the battery life test until we find a solution to a recurring
error on Chrome v89.x and later. The error prevents the test from completing
and producing a battery life estimate. Sometimes, the CrXPRT battery life test stops
running after only a few workload iterations, while at other times, it almost
reaches completion before producing the error.
We are cautiously optimistic that we’ve identified both the problem and a potential fix. We believe the problem stems from fluctuations in the time it takes the benchmark to communicate with Chrome to collect and store battery life information. While we haven’t identified the root cause of the fluctuations, adjusting the CrXPRT code to make it less sensitive to the fluctuations appears to be an effective fix. We have incorporated those adjustments into an updated, unpublished version of the app package, and we can now complete CrXPRT 2 battery life tests on Chrome v89.x and later with no failures.
are calling this a potential fix because we’re still testing across several
different Chromebook models to ensure consistency. In some testing, the
variance in estimated battery life results has been a little higher than we
like, so we’re taking time to determine whether that variance is present across
all systems or on only specific hardware.
We’d like to apologize once again for the inconvenience that this error is causing CrXPRT 2 testers. As soon as we better understand the viability of the current fix as a long-term update, we’ll let you know!
A few weeks ago, we discussed an error that we’d recently started encountering during the CrXPRT 2 battery life test on systems running Chrome OS v89.x and later.
The error prevents the test from completing and producing a battery life
estimate. CrXPRT stops running its normal workload cycle and produces a “Test
Error” page. The timing of the error can vary from run to run. Sometimes,
CrXPRT stops running after only a few workload iterations, while other times,
the battery life test almost reaches completion before producing the error.
We have seen the error on across multiple brands of Chromebooks running
Chrome OS v89.x and later. To our knowledge, Chromebooks running Chrome OS v88.x
and earlier versions complete the battery life test without issues. We are unaware
of any problems with the CrXPRT 2 performance test.
We’re continuing to investigate this problem. Unfortunately, we have not yet identified the root cause. Without a solution, we are recommending that for now, testers not use the CrXPRT 2 battery life test. We will post this recommendation on CrXPRT.com.
We apologize for the inconvenience that this error is causing CrXPRT 2 testers. As soon as we identify a possible solution, we will share that information here in the blog. If you have any insight into recent Chrome OS changes or flag settings that could be causing this problem, please let us know!
In recent lab tests, we’ve encountered an error during the CrXPRT 2 battery life test that prevents the test from completing and producing a battery life estimate. As the screenshot below shows, when the error occurs, CrXPRT stops running its normal workload cycle and produces a “Test Error” page. We have seen this behavior on systems running Chrome OS v89.x and v90.x, across multiple vendor platforms. In our testing, Chromebooks running Chrome OS v88.x and earlier versions continue to complete the battery life test without any issues.
The error occurs consistently on every Chromebook running v89.x or
v90.x that we’ve tested so far. However, the timing of the error varies from
run to run on the same system. Sometimes, CrXPRT stops running after only a few
workload iterations, while at other times, the battery life test runs almost to
completion before producing the error.
We’re actively investigating this problem, but have not yet
identified the root cause. We apologize for the inconvenience that this error
may be causing CrXPRT 2 testers. As soon as we identify the root cause of the
problem and have ideas about possible solutions, we will share that information
here in the blog. If you have any insight into recent Chrome OS changes or flag
settings that could be causing this problem, please let us know!
CrXPRT testers may
remember that back around the time that we began the CrXPRT 2 development process, the Chrome team announced that they were
phasing out support for Portable Native Client (PNaCL) in favor of WebAssembly (WASM). As a first step,
they changed the Chrome OS setting that enabled PNaCL by default. At the time,
this caused problems with the Photo Collage workload in CrXPRT 2015, and even
though we identified a workaround, details in the Chrome team’s announcement led us to conclude
that the workaround might stop working in June 2021. Because of this change, we
decided that the best decision would be to remove the workload from CrXPRT
2, and keep existing CrXPRT 2015 testers informed of any changes with the
In 2020, the Chrome
team also announced that they would be phasing out support for Chrome Apps
altogether starting in June 2021, and would shift their focus to Chrome
extensions. This change would have required us to reassess the viability of
CrXPRT in anything like its current form.
We’re happy to report that
the Chrome team has extended support for PNaCL and existing Chrome Apps through
June 2022. Barring further changes, this means that CrXPRT
2015 (with the workaround) and CrXPRT 2 should continue to serve as reliable
Chrome OS evaluation tools for some time.
If you have any questions about CrXPRT 2, please let us know!