BenchmarkXPRT Blog banner

Category: WebAssembly

The WebXPRT 4 results viewer is live!

In October, we shared an early preview of the new results viewer tool that we’ve been developing in parallel with WebXPRT 4. The WebXPRT 4 Preview is now available to the public, and we’re excited to announce that the new results viewer is also live. We already have over 65 test results in the viewer, and in the weeks leading up to the WebXPRT 4 general release, we’ll be actively populating the viewer with the latest PT-curated WebXPRT 4 Preview results.

We encourage readers to visit the blog for details about the viewer’s features, and to take some time to explore the data. We’re excited about this new tool, which we view as an ongoing project with room for expansion and improvement based on user feedback.

If you have any questions or comments about the WebXPRT 4 Preview or the new results viewer, please feel free to contact us!

Justin

Here’s what to expect in the WebXPRT 4 Preview

A few months ago, we shared detailed information about the changes we expected to make in WebXPRT 4. We are currently doing internal testing of the WebXPRT 4 Preview build in preparation for releasing it to the public. We want to let our readers know what to expect.

We’ve made some changes since our last update and some of the details we present below could still change before the preview release. However, we are much closer to the final product. Once we release the WebXPRT 4 Preview, testers will be able to publish scores from Preview build testing. We will limit any changes that we make between the Preview and the final release to the UI or features that are not expected to affect test scores.

General changes

Some of the non-workload changes we’ve made in WebXPRT 4 relate to our typical benchmark update process.

  • We have updated the aesthetics of the WebXPRT UI to make WebXPRT 4 visually distinct from older versions. We did not significantly change the flow of the UI.
  • We have updated content in some of the workloads to reflect changes in everyday technology, such as upgrading most of the photos in the photo processing workloads to higher resolutions.
  • We have not yet added a looping function to the automation scripts, but are still considering it for the future.
  • We investigated the possibility of shortening the benchmark by reducing the default number of iterations from seven to five, but have decided to stick with seven iterations to ensure that score variability remains acceptable across all platforms.

Workload changes

  • Photo Enhancement. We increased the efficiency of the workload’s Canvas object creation function, and replaced the existing photos with new, higher-resolution photos.
  • Organize Album Using AI. We replaced ConvNetJS with WebAssembly (WASM) based OpenCV.js for both the face detection and image classification tasks. We changed the images for the image classification tasks to images from the ImageNet dataset.
  • Stock Option Pricing. We updated the dygraph.js library.
  • Sales Graphs. We made no changes to this workload.
  • Encrypt Notes and OCR Scan. We replaced ASM.js with WASM for the Notes task and updated the WASM-based Tesseract version for the OCR task.
  • Online Homework. In addition to the existing scenario which uses four Web Workers, we have added a scenario with two Web Workers. The workload now covers a wider range of Web Worker performance, and we calculate the score by using the combined run time of both scenarios. We also updated the typo.js library.

Experimental workloads

As part of the WebXPRT 4 development process, we researched the possibility of including two new workloads: a natural language processing (NLP) workload, and an Angular-based message scrolling workload. After much testing and discussion, we have decided to not include these two workloads in WebXPRT 4. They will be good candidates for us to add as experimental WebXPRT 4 workloads in 2022.

The release timeline

Our goal is to publish the WebXPRT 4 preview build by December 15th, which will allow testers to publish scores in the weeks leading up to the Consumer Electronics Show in Las Vegas in January 2022. We will provide more detailed information about the GA timeline here in the blog as soon as possible.

If you have any questions about the details we’ve shared above, please feel free to ask!

Justin

A clearer picture of WebXPRT 4

The WebXPRT 4 development process is far enough along that we’d like to share more about changes we are likely to make and a rough target date for publishing a preview build. While some of the details below will probably change, this post should give readers a good sense of what to expect.

General changes

Some of the non-workload changes in WebXPRT 4 relate to our typical benchmark update process, and a few result directly from feedback we received from the WebXPRT tech press survey.

  • We will update the aesthetics of the WebXPRT UI to make WebXPRT 4 visually distinct from older versions. We do not anticipate significantly changing the flow of the UI.
  • We will update content in some of the workloads to reflect changes in everyday technology. For instance, we will upgrade most of the photos in the photo processing workloads to higher resolutions.
  • In response to a request from tech press survey respondents, we are considering adding a looping function to the automation scripts.
  • We are investigating the possibility of shortening the benchmark by reducing the default number of iterations from seven to five. We will only make this change if we can ensure that five iterations produce consistently low score variance.

Changes to existing workloads

  • Photo Enhancement. This workload applies three effects to two photos each (six photos total). It tests HTML5 Canvas, Canvas 2D, and JavaScript performance. The only change we are considering is adding higher-resolution photos.
  • Organize Album Using AI. This workload currently uses the ConvNetJS neural network library to complete two tasks: (1) organizing five images and (2) classifying the five images in an album. We are planning to replace ConvNetJS with WebAssembly (WASM) for both tasks and are considering upgrading the images to higher resolutions.
  • Stock Option Pricing. This workload calculates and displays graphic views of a stock portfolio using Canvas, SVG, and dygraph.js. The only change we are considering is combining it with the Sales Graphs workload (below).
  • Sales Graphs. This workload provides a web-based application displaying multiple views of sales data. Sales Graphs exercises HTML5 Canvas and SVG performance. The only change we are considering is combining it with the Stock Option Pricing workload (above).
  • Encrypt Notes and OCR Scan. This workload uses ASM.js to sync notes, extract text from a scanned receipt using optical character recognition (OCR), and add the scanned text to a spending report. We are planning to replace ASM.js with WASM for the Notes task and with WASM-based Tesseract for the OCR task.
  • Online Homework. This workload uses regex, arrays, strings, and Web Workers to review DNA and spell-check an essay. We are not planning to change this workload.

Possible new workloads

  • Natural Language Processing (NLP). We are considering the addition of an NLP workload using ONNX Runtime and/or TensorFlowJS. The workload would use Bidirectional Encoder Representations from Transformers (BERT) to answer questions about a given text. Similar use cases are becoming more prevalent in conversational bot systems, domain-specific document search tools, and various other educational applications.
  • Message Scrolling. We are considering developing a new workload that would use an Angular or React.js to scroll through hundreds of messages. We’ll share more about this possible workload as we firm up the details.

The release timeline

We hope to publish a WebXPRT 4 preview build in the second half of November, with a general release before the end of the year. If it looks as though that timeline will change significantly, we’ll provide an update here in the blog as soon as possible.

We’re very grateful for all the input we received during the WebXPRT 4 planning process. If you have any questions about the details we’ve shared above, please feel free to ask!

Justin

Round 2 of the WebXPRT 4 survey is now open

In May, we surveyed longtime WebXPRT users regarding the types of changes they would like to see in a WebXPRT 4. We sent the survey to journalists at several tech press outlets, and invited our blog readers to participate as well. We received some very helpful feedback. As we explore new possibilities for WebXPRT 4, we’ve decided to open an updated version of the survey. We’ve adjusted the questions a bit based on previous feedback and added some new ones, so we invite you to respond even if you participated in the original survey.

To do so, please send your answers to the following questions to benchmarkxprtsupport@principledtechnologies.com before July 31.

  • Do you think WebXPRT 3’s selection of workload scenarios is representative of modern web tasks?
  • How do you think WebXPRT compares to other common browser-based benchmarks, such as JetStream, Speedometer, and Octane?
  • Would you like to see a workload based on WebAssembly (WASM) in WebXPRT 4? Why or why not?
  • Would you like to see a workload based on Single Page Application (SPA) technology in WebXPRT 4? Why or why not?
  • Would you like to see a workload based on Motion UI in WebXPRT 4? Why or why not?
  • Would you like to see us include any other web technologies in additional workloads?
  • Are you happy with the WebXPRT 3 user interface? If not, what UI changes would you like to see?
  • Have you ever experienced significant connection issues when testing with WebXPRT?
  • Given its array of workloads, do you think the WebXPRT runtime is reasonable? Would you mind if the average runtime increased slightly?
  • Would you like to see us change any other aspects of WebXPRT 3?


If you would like to share your thoughts on any topics that the questions above do not cover, please include those in your response. We look forward to hearing from you!

Justin

Feedback from the WebXPRT 4 tech press survey

In early May, we sent a survey to members of the tech press who regularly use WebXPRT in articles and reviews. We asked for their thoughts on several aspects of WebXPRT, as well as what they’d like to see in the upcoming fourth version of the benchmark. We also published the survey questions here in the blog, and invited experienced WebXPRT testers to send their feedback as well. We received some good responses to the survey, and for the benefit of our readers, we’ve summarized some of the key comments and suggestions below.

  • One respondent stated that WebXPRT is demanding enough to test performance, but if we want to simulate modern web usage, we should find the most up-to-date studies on common browser tasks and web technologies. This suggestion lines up with our intention to study the feasibility of adding a WebAssembly workload
  • One respondent liked that fact that unlike many other browser benchmarks, WebXPRT tests more than just JavaScript calculation speed.
  • One respondent suggested that we include a link to a WebXPRT white paper within the UI, or at least a guide describing what happens during each workload.
  • One respondent stated that they would like for WebXPRT to automatically produce a good result file on the local test system.
  • One respondent said that WebXPRT has a relatively long runtime for a browser benchmark, and they would prefer that the runtime not increase in WebXPRT 4.
  • We had no direct calls for a battery life test, because many testers already have scripts and/or methodologies in place for battery testing, but one tester suggested adding the ability to loop the test so users can measure performance over varying lengths of time.
  • There were no requests to bring back any aspects of WebXPRT 2015 that we removed in WebXPRT 3.
  • There were no reports of significant connection issues when testing with WebXPRT.

We greatly appreciate the members of the tech press that responded to the survey. We’re still in the planning stages of WebXPRT 4, so there’s still time for anyone to send comments or ideas to benchmarkxprtsupport@principledtechnologies.com. We look forward to hearing from you!

Justin

The WebXPRT 4 tech press feedback survey

Device reviews in publications such as AnandTech, Notebookcheck, and PCMag, among many others, often feature WebXPRT test results, and we appreciate the many members of the tech press that use WebXPRT. As we move forward with the WebXPRT 4 development process, we’re especially interested in learning what longtime users would like to see in a new version of the benchmark.  

In previous posts, we’ve asked people to weigh in on the potential addition of a WebAssembly workload or a battery life test. We’d also like to ask experienced testers some other test-related questions. To that end, this week we’ll be sending a WebXPRT 4 survey directly to members of the tech press who frequently publish WebXPRT test results.

Regardless of whether you are a member of the tech press, we invite you to participate by sending your answers to any or all the questions below to benchmarkxprtsupport@principledtechnologies.com. We ask you to do so by the end of May.

  • Do you think WebXPRT 3’s selection of workload scenarios is representative of modern web tasks?
  • How do you think WebXPRT compares to other common browser-based benchmarks, such as JetStream, Speedometer, and Octane?
  • Are there web technologies that you’d like us to include in additional workloads?
  • Are you happy with the WebXPRT 3 user interface? If not, what UI changes would you like to see?
  • Are there any aspects of WebXPRT 2015 that we changed in WebXPRT 3 that you’d like to see us change back?
  • Have you ever experienced significant connection issues when testing with WebXPRT?
  • Given the array of workloads, do you think the WebXPRT runtime is reasonable? Would you mind if the average runtime were a bit longer?
  • Are there any other aspects of WebXPRT 3 that you’d like to see us change?

If you’d like to discuss any topics that we did not cover in the questions above, please feel free to include additional comments in your response. We look forward to hearing your thoughts!

Justin

Check out the other XPRTs:

Forgot your password?