BenchmarkXPRT Blog banner

Tag Archives: WASM

Web APIs: Possible paths for the AI-focused WebXPRT 4 auxiliary workload

In our last blog post, we discussed one of the major decision points we’re facing as we work on what we hope will be the first new AI-focused WebXPRT 4 auxiliary workload: choosing a Web AI framework. In today’s blog, we’re discussing another significant decision that we need to make for the future workload’s development path: choosing a web API.

Many of you are familiar with the concept of an application programming interface (API). Simply put, APIs implement sets of software rules, tools, and/or protocols that serve as intermediaries that make it possible for different computer programs or components to communicate with each other. APIs simplify many development tasks for programmers and provide standardized ways for applications to share data, functions, and system resources.

Web APIs fulfill the intermediary role of an API—through HTTP-based communication—for web servers (on the server side) or web browsers (on the client side). Client-side web APIs make it possible for browser-based applications to expand browser functionality. They execute the kinds of JavaScript, HTML5, and WebAssembly (Wasm) workloads—among other examples—that support the wide variety of browser extensions many of us use every day. WebXPRT uses those types of browser-based workloads to evaluate system performance. To lay a solid foundation for the first future browser-based AI workload, we need to choose a web API that will be compatible with WebXPRT and the Web AI framework and AI inference workload(s) we ultimately choose.

Currently, there are three main web API paths for running AI inference in a web browser: Web Neural Network (WebNN), Wasm, and WebGPU. These three web technologies are in various stages of development and standardization. Each has different levels of support within the major browsers. Here are basic overviews of each of the three options, as well as a few of our thoughts on the benefits and limitations that each may bring to the table for a future WebXPRT AI workload:

  • WebNN is a JavaScript API that enables developers to directly execute machine learning (ML) tasks on neural networks within web-based applications. WebNN makes it easier to integrate ML models into web apps, and it allows web apps to leverage the power of neural processing units (NPUs). WebNN has a lot going for it. It’s hardware-agnostic and works with various ML frameworks. It’s likely to be a major player in future browser-based inference applications. However, as a web standard, WebNN is still in the development stage and is only available in developer previews for Chromium-based browsers. Full default WebNN support could take a year or more.
  • Wasm is a binary instruction format that works across all modern browsers. Wasm provides a sandboxed environment that operates at near-native speeds and takes advantage of common hardware specs across platforms. Wasm’s capabilities offer web developers a great deal of flexibility for running complex client applications in the browser. Simply put, Wasm can help developers adapt their existing code for additional platforms and browser-based applications without requiring extensive code rewrites. Wasm’s flexibility and cross-platform compatibility is one of the reasons that we’ve already made use of Wasm in two existing WebXPRT 4 workloads that feature AI tasks: Organize Album using AI, and Encrypt Notes and OCR Scan. Wasm can also work together with other web APIs, such as WebGPU.
  • WebGPU enables web-based applications to directly access the graphics rendering and computational capabilities of a system’s GPU. The parallel computational abilities of GPUs make them especially well-suited to efficiently handle some of the demands of AI inference workloads, including image-based GenAI workloads or large language models. Google Chrome and Microsoft Edge currently support WebGPU, and it’s available in Safari through a tech preview.

Right now, we don’t think that WebNN will be fully out of the development phase in time to serve as our go-to web API for a new WebXPRT AI workload. Wasm and/or WebGPU appear to our best options for now. When WebNN is fully baked and available in mainstream browsers, it’s possible that we could port any existing Wasm- or WebGPU-based WebXPRT AI workloads to WebNN, which may open the possibility of cross-platform browser-based NPU performance comparisons.

All that said and as we mentioned in our previous post about Web AI frameworks, we have not made any final decisions about a web API or any aspect of the future workload. We’re still in the early stages of this project. We want your input.

If this discussion has sparked web AI ideas that you think would benefit the process, or if you have feedback you’d like to share, please feel free to contact us!

Justin

Making progress with WebXPRT 4 in iOS 17

In recent blog posts, we discussed an issue that we encountered when attempting to run WebXPRT 4 on iOS 17 devices. If you missed those posts, you can find more details about the nature of the problem here. In short, the issue is that the Encrypt Notes and OCR scan subtest in WebXPRT 4 gets stuck when the Tesseract.js Optical Character Recognition (OCR) engine attempts to scan a shopping receipt. We’ve verified that the issue occurs on devices running iOS 17, iPadOS 17, and macOS Sonoma with Safari 17.

After a good bit of troubleshooting and research to try and identify the cause of the problem, we decided to build an updated version of WebXPRT 4 that uses a newer version of Tesseract for the OCR task. Aside from updating Tesseract in the new build, we aimed to change as little as possible. To try and maximize continuity, we’re still using the original input image for the receipt scanning task, and we decided to stick with using the WASM library instead of a WASM-SIMD library. Aside from a new version of tesseract.js, WebXPRT 4 version number updates, and updated documentation where necessary, all other aspects of WebXPRT 4 will remain the same.

We’re currently testing a candidate build of this new version on a wide array of devices. The results so far seem promising, but we want to complete our due diligence and make sure this is the best approach to solving the problem. We know that OEM labs and tech reviewers put a lot of time and effort into compiling databases of results, so we hope to provide a solution that minimizes results disruption and inconvenience for WebXPRT 4 users. Ideally, folks would be able to integrate scores from the new build without any questions or confusion about comparability.

We don’t yet have an exact release date for a new WebXPRT 4 build, but we can say that we’re shooting for the end of October. We appreciate everyone’s patience as we work towards the best possible solution. If you have any questions or concerns about an updated version of WebXPRT 4, please let us know.

Justin

An update on the issue with WebXPRT 4 in iOS 17

Recently, we informed XPRT blog readers that after updating Apple iPhones and iPads to iOS and iPadOS 17, respectively, we began to see WebXPRT 4 failures on those devices. In the Safari and Google Chrome browsers, WebXPRT 4 test runs were freezing while running the Encrypt Notes and OCR Scan workload. We were able to replicate the issue on every iOS/iPadOS 17 device we tested, and we also confirmed that WebXPRT 4 continues to run without issues on other non-iOS platforms.

Our team has been investigating the situation, and we’ve made some progress. It’s clear that the failed test runs are getting stuck when the WASM-based Tesseract.js Optical Character Recognition (OCR) engine attempts to scan a shopping receipt. During our research, we’ve discovered an issue when the current Tesseract.js engine runs on iOS 17. This issue is broader than WebXPRT 4, and the Tesseract team is aware of the problem. Future versions of iOS 17 or later versions of Tesseract.js may include fixes for the problem, but unfortunately, we don’t know whether or when a fix will be available.

We’re currently investigating possible workarounds for the problem, and hope to be able to start testing soon. Our goal is that any solution we implement will not significantly affect existing WebXPRT 4 scores on non-iOS 17 platforms.

We will continue to share any substantive progress updates with readers here in the blog. Once again, we apologize for any inconvenience this issue causes for WebXPRT 4 users, and we appreciate your patience while we work toward a solution. If you have any questions or comments, please feel free to contact us!

Justin

How to automate WebXPRT 4 testing

As the number of WebXPRT runs continues to grow, we realize many new WebXPRT users may be unfamiliar with all the features and capabilities of the benchmark. To help inform users about features that might facilitate their testing, we’ve decided to highlight a few WebXPRT features here in the blog. A few weeks ago, we discussed the multiple language options available in the WebXPRT 4 UI. This week, we look at WebXPRT 4 test automation.

WebXPRT 4 allows users to run scripts in an automated fashion. You can control the execution of WebXPRT 4 by appending parameters and values to the WebXPRT URL. Three parameters are available: testtype, tests, and result. Below, you’ll find a description of those parameters and instructions for utilizing automation.

Test type

The WebXPRT automation framework accounts for two test types: (1) the six core workloads and (2) any experimental workloads we might add in future builds. There are currently no experimental tests in WebXPRT 4, so always set the test type variable to 1.

  • Core tests: 1

Test scenario

This parameter lets you specify which tests to run by using the following codes:

  • Photo enhancement: 1
  • Organize album using AI: 2
  • Stock option pricing: 4
  • Encrypt notes and OCR scan using WASM: 8
  • Sales graphs: 16
  • Online homework: 32

To run a single individual test, use its code. To run multiple tests, use the sum of their codes. For example, to run Stocks (4) and Notes (8), use the sum of 12. To run all core tests, use 63, the sum of all the individual test codes (1 + 2 + 4 + 8 + 16 + 32 = 63).

Results format

This parameter lets you select the format of the results:

  • Display the result as an HTML table: 1
  • Display the result as XML: 2
  • Display the result as CSV: 3
  • Download the result as CSV: 4

To use the automation feature, start with the URL http://www.principledtechnologies.com/benchmarkxprt/webxprt/2021/wx4_build_3_7_3, append a question mark (?), and add the parameters and values separated by ampersands (&). For example, to run all the core tests and download the results, you would use the following URL: http://principledtechnologies.com/benchmarkxprt/webxprt/2021/wx4_build_3_7_3/auto.php?testtype=1&tests=63&result=4

We hope the WebXPRT automation features will make testing easier for you. If you have any questions about WebXPRT or the automation process, please feel free to ask!

Justin

Considering WebAssembly for WebXPRT 4

Earlier this month, we discussed a few of our ideas for possible changes in WebXPRT 4, including new web technologies that may work well in a browser benchmark. Today, we’re going to focus on one of those technologies, WebAssembly, in more detail.

WebAssembly (WASM) is a binary instruction format that works across all modern browsers. WASM provides a sandboxed environment that operates at native speeds and takes advantage of common hardware specs across platforms. WASM’s capabilities offer web developers a great deal of flexibility for running complex client applications in the browser. That level of flexibility may enable workload scenario options for WebXPRT 4 such as gaming, video editing, VR, virtual machines, and image recognition. We’re excited about those possibilities, but it remains to be seen which WASM use cases will meet the criteria we look for when considering new WebXPRT workloads, such as relevancy to real life, consistency and replicability, and the broadest-possible level of cross-browser support.

One WASM workload that we’re investigating is a web-based machine learning workload with TensorFlow for JavaScript (TensorFlow.js). TensorFlow.js offers pre-trained models for a wide variety of tasks, including image classification, object detection, sentence encoding, and natural language processing. TensorFlow.js originally used WebGL technology on the back end, but now it’s possible to run the workload using WASM. We could also use this technology to enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.

We’re can’t yet say that a WASM workload will definitely appear in WebXPRT 4, but the technology is promising. Do you have any experience with WASM, or ideas for WASM workloads? There’s still time for you to help shape the future of WebXPRT 4, so let us know what you think!

Justin

Potential web technology additions for WebXPRT 4

A few months ago, we invited readers to send in their thoughts and ideas about web technologies and workload scenarios that may be a good fit for the next WebXPRT. We’d like to share a few of those ideas today, and we invite you to continue to send your feedback. We’re approaching the time when we need to begin firming up plans for a WebXPRT 4 development cycle in 2021, but there’s still plenty of time for you to help shape the future of the benchmark.

One of the most promising ideas for WebXPRT 4 is the potential addition of one or more WebAssembly (WASM) workloads. WASM is a low-level, binary instruction format that works across all modern browsers. It offers web developers a great deal of flexibility and provides the speed and efficiency necessary for running complex client applications in the browser. WASM enables a variety of workload scenario options, including gaming, video editing, VR, virtual machines, image recognition, and interactive educational content.

In addition, the Chrome team is dropping Portable Native Client (PNaCL) support in favor of WASM, which is why we had to remove a PNaCL workload when updating CrXPRT 2015 to CrXPRT 2. We generally model CrXPRT workloads on existing WebXPRT workloads, so familiarizing ourselves with WASM could ultimately benefit more than one XPRT benchmark.

We are also considering adding a web-based machine learning workload with TensorFlow for JavaScript (TensorFlow.js). TensorFlow.js offers pre-trained models for a wide variety of tasks including image classification, object detection, sentence encoding, natural language processing, and more. We could also use this technology to enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.

Other ideas include using a WebGL-based workload to target GPUs and investigating ways to incorporate a battery life test. What do you think? Let us know!

Justin

Check out the other XPRTs:

Forgot your password?