BenchmarkXPRT Blog banner

Tag Archives: battery life

Persistent CrXPRT 2 battery life test error on Chrome v89 and later

A few weeks ago, we discussed an error that we’d recently started encountering during the CrXPRT 2 battery life test on systems running Chrome OS v89.x and later. The error prevents the test from completing and producing a battery life estimate. CrXPRT stops running its normal workload cycle and produces a “Test Error” page. The timing of the error can vary from run to run. Sometimes, CrXPRT stops running after only a few workload iterations, while other times, the battery life test almost reaches completion before producing the error.

We have seen the error on across multiple brands of Chromebooks running Chrome OS v89.x and later. To our knowledge, Chromebooks running Chrome OS v88.x and earlier versions complete the battery life test without issues. We are unaware of any problems with the CrXPRT 2 performance test.

We’re continuing to investigate this problem. Unfortunately, we have not yet identified the root cause. Without a solution, we are recommending that for now, testers not use the CrXPRT 2 battery life test. We will post this recommendation on CrXPRT.com.

We apologize for the inconvenience that this error is causing CrXPRT 2 testers. As soon as we identify a possible solution, we will share that information here in the blog. If you have any insight into recent Chrome OS changes or flag settings that could be causing this problem, please let us know!

Justin

Feedback from the WebXPRT 4 tech press survey

In early May, we sent a survey to members of the tech press who regularly use WebXPRT in articles and reviews. We asked for their thoughts on several aspects of WebXPRT, as well as what they’d like to see in the upcoming fourth version of the benchmark. We also published the survey questions here in the blog, and invited experienced WebXPRT testers to send their feedback as well. We received some good responses to the survey, and for the benefit of our readers, we’ve summarized some of the key comments and suggestions below.

  • One respondent stated that WebXPRT is demanding enough to test performance, but if we want to simulate modern web usage, we should find the most up-to-date studies on common browser tasks and web technologies. This suggestion lines up with our intention to study the feasibility of adding a WebAssembly workload
  • One respondent liked that fact that unlike many other browser benchmarks, WebXPRT tests more than just JavaScript calculation speed.
  • One respondent suggested that we include a link to a WebXPRT white paper within the UI, or at least a guide describing what happens during each workload.
  • One respondent stated that they would like for WebXPRT to automatically produce a good result file on the local test system.
  • One respondent said that WebXPRT has a relatively long runtime for a browser benchmark, and they would prefer that the runtime not increase in WebXPRT 4.
  • We had no direct calls for a battery life test, because many testers already have scripts and/or methodologies in place for battery testing, but one tester suggested adding the ability to loop the test so users can measure performance over varying lengths of time.
  • There were no requests to bring back any aspects of WebXPRT 2015 that we removed in WebXPRT 3.
  • There were no reports of significant connection issues when testing with WebXPRT.

We greatly appreciate the members of the tech press that responded to the survey. We’re still in the planning stages of WebXPRT 4, so there’s still time for anyone to send comments or ideas to benchmarkxprtsupport@principledtechnologies.com. We look forward to hearing from you!

Justin

CrXPRT 2 battery life error on Chrome 89 and 90

In recent lab tests, we’ve encountered an error during the CrXPRT 2 battery life test that prevents the test from completing and producing a battery life estimate. As the screenshot below shows, when the error occurs, CrXPRT stops running its normal workload cycle and produces a “Test Error” page. We have seen this behavior on systems running Chrome OS v89.x and v90.x, across multiple vendor platforms. In our testing, Chromebooks running  Chrome OS v88.x and earlier versions continue to complete the battery life test without any issues.

The error occurs consistently on every Chromebook running v89.x or v90.x that we’ve tested so far. However, the timing of the error varies from run to run on the same system. Sometimes, CrXPRT stops running after only a few workload iterations, while at other times, the battery life test runs almost to completion before producing the error.

We’re actively investigating this problem, but have not yet identified the root cause. We apologize for the inconvenience that this error may be causing CrXPRT 2 testers. As soon as we identify the root cause of the problem and have ideas about possible solutions, we will share that information here in the blog. If you have any insight into recent Chrome OS changes or flag settings that could be causing this problem, please let us know!

Justin

Considering a battery life test for WebXPRT 4

A few weeks ago, we discussed the beginnings of a WebXPRT 4 development plan, and asked for reader feedback about potential workload changes. So far, the two most common feedback topics have been the possible addition of a WebAssembly workload, and the feasibility of including a browser-based battery life test. Today, we discuss what a WebXPRT 4 battery life test would look like, and some of the challenges we’d have to overcome to make it a reality.

Battery life tests fall into two primary categories: simple rundown tests and performance-weighted tests. Simple rundown tests measure battery life during extreme idle periods and loops of movie playbacks, etc., but do not reflect the wide-ranging mix of activities that characterize a typical day for most users. While they can be useful for performing very specific apple-to-apples comparisons, these tests have limited value when it comes to giving consumers a realistic estimation of the battery life they would experience during everyday use.

In contrast, performance-weighted battery life tests, such as the one in CrXPRT 2, attempt to reflect real-world usage. The CrXPRT battery life test simulates common daily usage patterns for Chromebooks by including all the productivity workloads from the performance test, plus video playback, audio playback, and gaming scenarios. It also includes periods of wait/idle time. We believe this mixture of diverse activity and idle time better represents typical real-life behavior patterns. This makes the resulting estimated battery life much more helpful for consumers who are trying to match a device’s capabilities with their real-world needs.

From a technical standpoint, WebXPRT’s cross-platform nature presents us with several challenges that we did not face while developing the CrXPRT battery life test for Chrome OS. While the WebXPRT performance tests run in almost any browser, cross-browser differences in battery life reporting may restrict the battery life test to a single browser. For instance, Mozilla has deprecated the battery status API for Firefox, and we’re not yet sure if there’s another approach that would work. If a WebXPRT 4 battery life test supported only a single browser, such as Chrome or Safari, would you still be interested in using it? Please let us know.

A browser-based battery life workflow also presents other challenges that we do not face in native client applications such as CrXPRT:

  • A browser-based battery life test would require the user to check the starting and ending battery capacities, with no way for the app to independently verify data accuracy.
  • The battery life test could require more babysitting in the event of network issues. We can catch network failures and try to handle them by reporting periods of network disconnection, but those interruptions could influence the battery life duration.
  • The factors above could make it difficult to achieve repeatability. One way to address that problem would be to run the test in a standardized lab environment lab with a steady internet connection, but a long list of standardized environmental requirements would make the battery life test less attractive and less accessible to many testers.

Our intention with today’s blog is not to make a WebXPRT 4 battery life test seem like an impossibility. Rather, we want to share our perspective on what the test might look like, and describe some of the challenges and considerations in play. If you have thoughts about battery life testing, or experience with battery life APIs in one or more of the major browsers, we’d love to hear from you!

Justin

Moving forward with WebXPRT 4

In the coming months, we’ll be moving forward with the first stages of the WebXPRT 4 development process. It’s been a while since we last asked readers to send their thoughts about web technologies and workloads that may be a good fit for WebXPRT 4, but we’re still very much open to ideas. If you missed our previous posts about possible changes for WebXPRT 4, we recap the most prominent ideas below. We also request specific feedback regarding a potential battery life component.

  • Community members have asked about a WebXPRT 4 battery life test. Any such test would likely be very similar to the performance-weighted battery life test in CrXPRT 2 (as opposed to a simple rundown test). While WebXPRT runs in almost any browser, cross-browser compatibility issues could cause a WebXPRT battery life test to run in only one browser. If this turned out to be the case, would you still be interested in using the battery life test? Please let us know.
  • One of the most promising ideas is the potential addition of one or more WebAssembly (WASM) workloads. WASM is a low-level, binary instruction format that works across all modern browsers. It offers web developers a great deal of flexibility and provides the speed and efficiency necessary for running complex client applications in the browser. WASM enables a variety of workload scenario options, including gaming, video editing, VR, virtual machines, image recognition, and interactive educational content.
  • We are also considering adding a web-based machine learning workload with TensorFlow for JavaScript (TensorFlow.js). TensorFlow.js offers pre-trained models for a wide range of tasks including image classification, object detection, sentence encoding, and natural language processing. We could also use this technology to enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.
  • Other ideas include using a WebGL-based workload to target GPUs, and simulating common web applications.

We’ll start work on WebXPRT 4 soon, but there’s still time to send your comments and ideas, so please do so as quickly as possible!

Justin

Following up on CrXPRT 2 battery life errors

A few weeks ago, we discussed error messages that a tester received when starting up CrXPRT 2 after a battery life test. CrXPRT 2 battery life tests require a full battery rundown, after which the tester plugs in the Chromebook, turns it on, opens the CrXPRT 2 app, and sees the test results. In the reported cases, the tester opened the app after a battery life test that seemed successful, but saw “N/A” or “test error” messages instead of the results they expected.

During discussions about the end-of-test system environment, we realized that some testers might be unclear about how to tell that the battery has fully run down. During the system idle portion of CrXPRT 2 battery life test iterations, the Chromebook screen turns black and a small cursor appears somewhere on the screen to let testers know the test is still in progress. We believe that some testers, seeing the black screen but not the cursor, believe the system has shut down. Restarting CrXPRT 2 before the battery life test is complete could explain some of the “N/A” or “test error” messages users have reported.

If you see a black screen without a cursor, you can check to see whether the test is complete by looking for the small system power indicator light on the side or top of most Chromebooks. These are usually red, orange, or green, but if a light of any color is lit, the test is still underway. When the light goes out, the test has ended. You can plug the system in and power it on to see results.

Note that some Chromebooks provide low-battery warnings onscreen. During CrXPRT 2 battery life runs, testers should ignore these.

We hope this clears up any confusion about how to know when a CrXPRT 2 battery life test has ended. If you’ve received repeated “N/A” or “test error” messages during your CrXPRT 2 testing and the information above does not help, please let us know!

Justin

Check out the other XPRTs:

Forgot your password?