BenchmarkXPRT Blog banner

Search Results for: webxprt

The WebXPRT 4 tech press feedback survey

Device reviews in publications such as AnandTech, Notebookcheck, and PCMag, among many others, often feature WebXPRT test results, and we appreciate the many members of the tech press that use WebXPRT. As we move forward with the WebXPRT 4 development process, we’re especially interested in learning what longtime users would like to see in a new version of the benchmark.  

In previous posts, we’ve asked people to weigh in on the potential addition of a WebAssembly workload or a battery life test. We’d also like to ask experienced testers some other test-related questions. To that end, this week we’ll be sending a WebXPRT 4 survey directly to members of the tech press who frequently publish WebXPRT test results.

Regardless of whether you are a member of the tech press, we invite you to participate by sending your answers to any or all the questions below to benchmarkxprtsupport@principledtechnologies.com. We ask you to do so by the end of May.

  • Do you think WebXPRT 3’s selection of workload scenarios is representative of modern web tasks?
  • How do you think WebXPRT compares to other common browser-based benchmarks, such as JetStream, Speedometer, and Octane?
  • Are there web technologies that you’d like us to include in additional workloads?
  • Are you happy with the WebXPRT 3 user interface? If not, what UI changes would you like to see?
  • Are there any aspects of WebXPRT 2015 that we changed in WebXPRT 3 that you’d like to see us change back?
  • Have you ever experienced significant connection issues when testing with WebXPRT?
  • Given the array of workloads, do you think the WebXPRT runtime is reasonable? Would you mind if the average runtime were a bit longer?
  • Are there any other aspects of WebXPRT 3 that you’d like to see us change?

If you’d like to discuss any topics that we did not cover in the questions above, please feel free to include additional comments in your response. We look forward to hearing your thoughts!

Justin

WebXPRT passes the 750,000-run milestone!

We’re excited to see that users have successfully completed over 750,000 WebXPRT runs! If you’ve run WebXPRT in any of the more than 654 cities and 68 countries from which we’ve received complete test data—including newcomers Belize, Cambodia, Croatia, and Pakistan—we’re grateful for your help. We could not have reached this milestone without you!

As the chart below illustrates, WebXPRT use has grown steadily over the years. We now record, on average, almost twice as many WebXPRT runs in one month as we recorded in the entirety of our first year. In addition, with over 82,000 runs to date in 2021, there are no signs that growth is slowing.

Developing a new benchmark is never easy, and the obstacles multiply when you attempt to create a cross-platform benchmark, such as WebXPRT, that will run on a wide variety of devices. Establishing trust with the benchmarking community is another challenge. Transparency, consistency, and technical competency on our part are critical factors in building that trust, but the people who take time out of their busy schedules to run the benchmark for the first time also play a role. We thank all of the manufacturers, OEM labs, and members of the tech press who decided to give WebXPRT a try, and we look forward to your input as we continue to improve WebXPRT in the years to come. 

If you have any questions or comments about WebXPRT, we’d love to hear from you!

Justin

Considering a battery life test for WebXPRT 4

A few weeks ago, we discussed the beginnings of a WebXPRT 4 development plan, and asked for reader feedback about potential workload changes. So far, the two most common feedback topics have been the possible addition of a WebAssembly workload, and the feasibility of including a browser-based battery life test. Today, we discuss what a WebXPRT 4 battery life test would look like, and some of the challenges we’d have to overcome to make it a reality.

Battery life tests fall into two primary categories: simple rundown tests and performance-weighted tests. Simple rundown tests measure battery life during extreme idle periods and loops of movie playbacks, etc., but do not reflect the wide-ranging mix of activities that characterize a typical day for most users. While they can be useful for performing very specific apple-to-apples comparisons, these tests have limited value when it comes to giving consumers a realistic estimation of the battery life they would experience during everyday use.

In contrast, performance-weighted battery life tests, such as the one in CrXPRT 2, attempt to reflect real-world usage. The CrXPRT battery life test simulates common daily usage patterns for Chromebooks by including all the productivity workloads from the performance test, plus video playback, audio playback, and gaming scenarios. It also includes periods of wait/idle time. We believe this mixture of diverse activity and idle time better represents typical real-life behavior patterns. This makes the resulting estimated battery life much more helpful for consumers who are trying to match a device’s capabilities with their real-world needs.

From a technical standpoint, WebXPRT’s cross-platform nature presents us with several challenges that we did not face while developing the CrXPRT battery life test for Chrome OS. While the WebXPRT performance tests run in almost any browser, cross-browser differences in battery life reporting may restrict the battery life test to a single browser. For instance, Mozilla has deprecated the battery status API for Firefox, and we’re not yet sure if there’s another approach that would work. If a WebXPRT 4 battery life test supported only a single browser, such as Chrome or Safari, would you still be interested in using it? Please let us know.

A browser-based battery life workflow also presents other challenges that we do not face in native client applications such as CrXPRT:

  • A browser-based battery life test would require the user to check the starting and ending battery capacities, with no way for the app to independently verify data accuracy.
  • The battery life test could require more babysitting in the event of network issues. We can catch network failures and try to handle them by reporting periods of network disconnection, but those interruptions could influence the battery life duration.
  • The factors above could make it difficult to achieve repeatability. One way to address that problem would be to run the test in a standardized lab environment lab with a steady internet connection, but a long list of standardized environmental requirements would make the battery life test less attractive and less accessible to many testers.

Our intention with today’s blog is not to make a WebXPRT 4 battery life test seem like an impossibility. Rather, we want to share our perspective on what the test might look like, and describe some of the challenges and considerations in play. If you have thoughts about battery life testing, or experience with battery life APIs in one or more of the major browsers, we’d love to hear from you!

Justin

Considering WebAssembly for WebXPRT 4

Earlier this month, we discussed a few of our ideas for possible changes in WebXPRT 4, including new web technologies that may work well in a browser benchmark. Today, we’re going to focus on one of those technologies, WebAssembly, in more detail.

WebAssembly (WASM) is a binary instruction format that works across all modern browsers. WASM provides a sandboxed environment that operates at native speeds and takes advantage of common hardware specs across platforms. WASM’s capabilities offer web developers a great deal of flexibility for running complex client applications in the browser. That level of flexibility may enable workload scenario options for WebXPRT 4 such as gaming, video editing, VR, virtual machines, and image recognition. We’re excited about those possibilities, but it remains to be seen which WASM use cases will meet the criteria we look for when considering new WebXPRT workloads, such as relevancy to real life, consistency and replicability, and the broadest-possible level of cross-browser support.

One WASM workload that we’re investigating is a web-based machine learning workload with TensorFlow for JavaScript (TensorFlow.js). TensorFlow.js offers pre-trained models for a wide variety of tasks, including image classification, object detection, sentence encoding, and natural language processing. TensorFlow.js originally used WebGL technology on the back end, but now it’s possible to run the workload using WASM. We could also use this technology to enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.

We’re can’t yet say that a WASM workload will definitely appear in WebXPRT 4, but the technology is promising. Do you have any experience with WASM, or ideas for WASM workloads? There’s still time for you to help shape the future of WebXPRT 4, so let us know what you think!

Justin

Moving forward with WebXPRT 4

In the coming months, we’ll be moving forward with the first stages of the WebXPRT 4 development process. It’s been a while since we last asked readers to send their thoughts about web technologies and workloads that may be a good fit for WebXPRT 4, but we’re still very much open to ideas. If you missed our previous posts about possible changes for WebXPRT 4, we recap the most prominent ideas below. We also request specific feedback regarding a potential battery life component.

  • Community members have asked about a WebXPRT 4 battery life test. Any such test would likely be very similar to the performance-weighted battery life test in CrXPRT 2 (as opposed to a simple rundown test). While WebXPRT runs in almost any browser, cross-browser compatibility issues could cause a WebXPRT battery life test to run in only one browser. If this turned out to be the case, would you still be interested in using the battery life test? Please let us know.
  • One of the most promising ideas is the potential addition of one or more WebAssembly (WASM) workloads. WASM is a low-level, binary instruction format that works across all modern browsers. It offers web developers a great deal of flexibility and provides the speed and efficiency necessary for running complex client applications in the browser. WASM enables a variety of workload scenario options, including gaming, video editing, VR, virtual machines, image recognition, and interactive educational content.
  • We are also considering adding a web-based machine learning workload with TensorFlow for JavaScript (TensorFlow.js). TensorFlow.js offers pre-trained models for a wide range of tasks including image classification, object detection, sentence encoding, and natural language processing. We could also use this technology to enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.
  • Other ideas include using a WebGL-based workload to target GPUs, and simulating common web applications.

We’ll start work on WebXPRT 4 soon, but there’s still time to send your comments and ideas, so please do so as quickly as possible!

Justin

Using WebXPRT 3 to compare the performance of popular browsers (Round 2)

It’s been nine months since we’ve published a WebXPRT 3 browser performance comparison, so we decided to put the newest versions of popular browsers through the paces to see if the performance rankings have changed since our last round of tests.

We used the same laptop as last time: a Dell XPS 13 7930 with an Intel Core i3-10110U processor and 4 GB of RAM running Windows 10 Home, updated to version 1909 (18363.1139). We installed all current Windows updates and tested on a clean system image. After the update process completed, we turned off updates to prevent them from interfering with test runs. We ran WebXPRT 3 three times on five browsers: Brave, Google Chrome, Microsoft Edge, Mozilla Firefox, and Opera. The posted score for each browser is the median of the three test runs.

In our last round of tests, the four Chromium-based browsers (Brave, Chrome, Edge, and Opera) produced scores that were nearly identical. Only Mozilla Firefox produced a significantly different (and better) score. The parity of the Chromium-based browsers was not surprising, considering they have the same underlying foundation.

In this round of testing, the Chromium-based browsers again produced very close scores, although Brave’s performance lagged by about 4 percent. Firefox again separated itself from the pack with a higher score. With the exception of Chrome, which produced an identical score as last time, every browser’s score was slightly slower than before. There are many possible reasons for this, including increased overhead in the browsers or changes in Windows, and the respective slowdowns for each browser will probably be unnoticeable to most users during everyday tasks.

Do these results mean that Mozilla Firefox will provide you with a speedier web experience? As we noted in the last comparison, a device with a higher WebXPRT score will probably feel faster during daily use than one with a lower score. For comparisons on the same system, however, the answer depends in part on the types of things you do on the web, how the extensions you’ve installed affect performance, how frequently the browsers issue updates and incorporate new web technologies, and how accurately each browsers’ default installation settings reflect how you would set up that browser for your daily workflow.

In addition, browser speed can increase or decrease significantly after an update, only to swing back in the other direction shortly thereafter. OS-specific optimizations can also affect performance, such as with Edge on Windows 10 and Chrome on Chrome OS. All of these variables are important to keep in mind when considering how browser performance comparison results translate to your everyday experience.

What are your thoughts on browser performance? Let us know!

Justin

Check out the other XPRTs:

Forgot your password?