BenchmarkXPRT Blog banner

Tag Archives: Safari

WebXPRT 4 is good to go with the latest Apple software release

Last month, we reported the good news that our WebXPRT 4 tests successfully ran to completion on the beta releases of iOS 17.2, iPadOS 17.2, and macOS Sonoma 14.2 with Safari 17.2. When we tested with those beta builds, WebXPRT 4 did not encounter the issue of test runs getting stuck on iOS 17.1 while attempting to complete the receipt scanning task in the Encrypt Notes and OCR Scan subtest. Unfortunately, during the past several weeks, this fix was only available to Apple users running beta software through the Apple Developer Program.

We’re happy to report that Apple has now finalized and published the general releases of iOS 17.2, iPadOS 17.2, and Safari 17.2. WebXPRT 4 tests running on those platforms should now complete without any problems.

We do appreciate everyone’s patience as we worked to find a solution to this problem, and we look forward to seeing your WebXPRT 4 scores from all the latest Apple devices! If you have any questions or concerns about WebXPRT 4, or you encounter any additional issues when running the test on any platform, please let us know.

Justin

Good news for WebXPRT 4 testing!

Over the past several weeks, we’ve been working to find a solution to a problem with WebXPRT 4 test failures on Apple devices running iOS 17/17.1, iPadOS 17/17.1, and macOS Sonoma with Safari 17/17.1. While we put significant effort into an updated WebXPRT version that would mitigate this issue, we are happy to report that it now looks like we’ll be able to stick with the current version!

Last Thursday, Apple released the iOS 17.2 beta for participants in the Apple Developer Program. When we tested the current version of WebXPRT 4 on iOS 17.2, the tests completed without any issues. We then successfully completed tests on iPadOS 17.2 and macOS Sonoma 14.2 with Safari 17.2. Now that we have good reasons to believe that the iOS 17.2 release will solve the problem, sticking with the current WebXPRT 4 build will maximize continuity and minimize disruption for WebXPRT users.

Apple has not yet published a public release date for iOS/iPad OS/Safari 17.2. Based on past development schedules, it seems likely that they will release it between mid-November and early December, but that’s simply our best guess. Until then, users who want to test WebXPRT 4 on devices running iOS 17/17.1, iPadOS 17/17.1, or macOS Sonoma with Safari 17/17.1 will need to update those devices to iOS/iPad OS/Safari 17.2 via the Apple Developer Program.

To help Apple users better navigate testing until the public 17.2 release, we’ve added a function to the current WebXPRT 4 start page that will notify users if they need to update their operating system to test.

We appreciate everyone’s patience as we worked to find a solution to this problem! If you have any questions or concerns about WebXPRT 4, please let us know.

Justin

An update on the issue with WebXPRT 4 in iOS 17

Recently, we informed XPRT blog readers that after updating Apple iPhones and iPads to iOS and iPadOS 17, respectively, we began to see WebXPRT 4 failures on those devices. In the Safari and Google Chrome browsers, WebXPRT 4 test runs were freezing while running the Encrypt Notes and OCR Scan workload. We were able to replicate the issue on every iOS/iPadOS 17 device we tested, and we also confirmed that WebXPRT 4 continues to run without issues on other non-iOS platforms.

Our team has been investigating the situation, and we’ve made some progress. It’s clear that the failed test runs are getting stuck when the WASM-based Tesseract.js Optical Character Recognition (OCR) engine attempts to scan a shopping receipt. During our research, we’ve discovered an issue when the current Tesseract.js engine runs on iOS 17. This issue is broader than WebXPRT 4, and the Tesseract team is aware of the problem. Future versions of iOS 17 or later versions of Tesseract.js may include fixes for the problem, but unfortunately, we don’t know whether or when a fix will be available.

We’re currently investigating possible workarounds for the problem, and hope to be able to start testing soon. Our goal is that any solution we implement will not significantly affect existing WebXPRT 4 scores on non-iOS 17 platforms.

We will continue to share any substantive progress updates with readers here in the blog. Once again, we apologize for any inconvenience this issue causes for WebXPRT 4 users, and we appreciate your patience while we work toward a solution. If you have any questions or comments, please feel free to contact us!

Justin

Investigating a possible issue with WebXPRT 4 in iOS 17

Yesterday, Apple revealed the iPhone 15 and iPhone 15 Pro at its annual fall event, along with a new version of the iOS mobile operating system (iOS 17). The official iOS 17 launch will take place on September 18th, but before then, users of newer iPhones can install the OS via the Apple Beta Software Program.

Today, a tech journalist informed us that during their testing of iPhone 15 and iPhone 15 Pro with iOS 17 Beta models, WebXPRT 4 has been freezing while running the Encrypt Notes and OCR Scan workload in the Safari 17 browser. Here in the lab, we were able to immediately replicate the issue on an iPhone 12 Pro with iOS 17 Beta model.

Our initial troubleshooting confirmed that WebXPRT 3 successfully runs to completion on iOS 17 Beta, so it appears that the problem is specific to WebXPRT 4. We also confirmed that WebXPRT 4 freezes at the same place when running in the Google Chrome browser on iOS 17 Beta, so we know that the problem does not occur only in Safari.

We’re currently investigating the issue, and will publish our findings here in the blog as soon as we feel confident that we’ve identified both the root cause and a workable solution, if a solution is necessary. One reason a solution would not be necessary is that the issue is a bug on the iOS 17 Beta side that Apple will resolve before the official launch.

We apologize for any inconvenience this issue might cause for tech reviewers and iPhone users, and we appreciate your patience while we figure out what’s going on. If you have any questions about WebXPRT 4, please don’t hesitate to ask!

Justin

Considering a battery life test for WebXPRT 4

A few weeks ago, we discussed the beginnings of a WebXPRT 4 development plan, and asked for reader feedback about potential workload changes. So far, the two most common feedback topics have been the possible addition of a WebAssembly workload, and the feasibility of including a browser-based battery life test. Today, we discuss what a WebXPRT 4 battery life test would look like, and some of the challenges we’d have to overcome to make it a reality.

Battery life tests fall into two primary categories: simple rundown tests and performance-weighted tests. Simple rundown tests measure battery life during extreme idle periods and loops of movie playbacks, etc., but do not reflect the wide-ranging mix of activities that characterize a typical day for most users. While they can be useful for performing very specific apple-to-apples comparisons, these tests have limited value when it comes to giving consumers a realistic estimation of the battery life they would experience during everyday use.

In contrast, performance-weighted battery life tests, such as the one in CrXPRT 2, attempt to reflect real-world usage. The CrXPRT battery life test simulates common daily usage patterns for Chromebooks by including all the productivity workloads from the performance test, plus video playback, audio playback, and gaming scenarios. It also includes periods of wait/idle time. We believe this mixture of diverse activity and idle time better represents typical real-life behavior patterns. This makes the resulting estimated battery life much more helpful for consumers who are trying to match a device’s capabilities with their real-world needs.

From a technical standpoint, WebXPRT’s cross-platform nature presents us with several challenges that we did not face while developing the CrXPRT battery life test for Chrome OS. While the WebXPRT performance tests run in almost any browser, cross-browser differences in battery life reporting may restrict the battery life test to a single browser. For instance, Mozilla has deprecated the battery status API for Firefox, and we’re not yet sure if there’s another approach that would work. If a WebXPRT 4 battery life test supported only a single browser, such as Chrome or Safari, would you still be interested in using it? Please let us know.

A browser-based battery life workflow also presents other challenges that we do not face in native client applications such as CrXPRT:

  • A browser-based battery life test would require the user to check the starting and ending battery capacities, with no way for the app to independently verify data accuracy.
  • The battery life test could require more babysitting in the event of network issues. We can catch network failures and try to handle them by reporting periods of network disconnection, but those interruptions could influence the battery life duration.
  • The factors above could make it difficult to achieve repeatability. One way to address that problem would be to run the test in a standardized lab environment lab with a steady internet connection, but a long list of standardized environmental requirements would make the battery life test less attractive and less accessible to many testers.

Our intention with today’s blog is not to make a WebXPRT 4 battery life test seem like an impossibility. Rather, we want to share our perspective on what the test might look like, and describe some of the challenges and considerations in play. If you have thoughts about battery life testing, or experience with battery life APIs in one or more of the major browsers, we’d love to hear from you!

Justin

Check out the other XPRTs:

Forgot your password?