BenchmarkXPRT Blog banner

Category: Performance testing on tablets

Support for MobileXPRT 3 will likely end soon

In a past blog post, we discussed our plan to move several older versions of XPRT benchmarks to an XPRT archive page. Some of those legacy XPRTs still function correctly, and testers occasionally use them, but a few no longer work on the latest versions of the operating systems or browsers that we designed them to test. With the archive page, we can prevent potential confusion for new users who visit current XPRT pages, but still provide longtime users with continued access to old tests.

You can find more information about the XPRTs that we’ll be moving to the archive page here, but today, we want to let MobileXPRT users know that there’s a high likelihood that MobileXPRT 3 will be joining the list of archived XPRTs in the very near future. The Google Play Store has notified us that, due to evolving requirements for apps in newer versions of Android, we must update our MobileXPRT 3 app package to target an Android API level within one year of the latest Android release. If we don’t update the app to meet that requirement by November 1, users will no longer be able to access MobileXPRT 3 through the Google Play Store.

Though a small number of labs and reviewers still use MobileXPRT 3 to test phones and tablets around the world, we don’t feel current usage is high enough for us to justify committing resources to an update at this point. We had hoped that even if MobileXPRT 3 became inaccessible via the Google Play Store, it would still be possible to sideload the app for testing on newer Android devices. After experimenting with installation options in the lab, however, we think it’s likely that settings on devices running Android 11 and up will prevent both Google Play and sideload installations after November 1. The situation may change, but right now, we don’t expect any method to work after that date. If you try, you’ll likely encounter a message during the installation process that says, “This app was built for an older version of Android and may not work properly. Try checking for updates, or contact the developer.” If you attempt to continue the installation process after that message appears, the app will crash.

Both Android and Chrome developers know that the respective stores sometimes extend these types of deadlines. We hope that will be the case here, but we have no information that would lead us to anticipate an extension. If there is no extension, we will still make MobileXPRT 3 available for testing on older Android devices, but we will then have to move it to the XPRT archive page.

We’re grateful for everyone who has used MobileXPRT 3 in the past, and we apologize for any convenience this change may cause. If you have any questions or concerns about MobileXPRT 3 access, please let us know

Justin

We’ve updated MobileXPRT 3 to address issues with Android 11

This week, we published an updated MobileXPRT 3 build, version 3.116.0.4, on MobileXPRT.com and in the Google Play Store. The new build addresses an issue we recently discovered, where MobileXPRT was crashing after installation on some Android 11 phones. Permissions requirements and a new storage strategy called scoped storage were causing the problem. By default, scoped storage restricts an app’s storage access to app-specific directories and media, and prohibits general access to external or public directories. It also prevents third-party apps such as email clients or file managers from accessing MobileXPRT 3 results files. This default setting requires an opt-in permissions prompt that MobileXPRT 3 did not have prior to this week’s release.

MobileXPRT 3.116.0.4 points all of the benchmark’s file references to its private directory and allows users to zip results files and attach them to results submission emails. Neither change affects the testing process or test scores. If you have any questions or comments about the new MobileXPRT 3 build, please let us know!

Justin

The XPRT Spotlight Black Friday Showcase helps you shop with confidence

Black Friday and Cyber Monday are almost here, and you may be feeling overwhelmed by the sea of tech gifts to choose from. The XPRTs are here to help. We’ve gathered the product specs and performance facts for some of the hottest tech devices in one convenient place—the XPRT Spotlight Black Friday Showcase. The Showcase is a free shopping tool that provides side-by-side comparisons of some of the season’s most popular smartphones, laptops, Chromebooks, tablets, and PCs. It helps you make informed buying decisions so you can shop with confidence this holiday season.

Want to know how the Google Pixel 4 stacks up against the Apple iPhone 11 or Samsung Galaxy Note10 in web browsing performance or screen size? Simply select any two devices in the Showcase and click Compare. You can also search by device type if you’re interested in a specific form factor such as consoles or tablets.

The Showcase doesn’t go away after Black Friday. We’ll rename it the XPRT Holiday Showcase and continue to add devices such as the Microsoft Surface Pro X throughout the shopping season. Be sure to check back in and see how your tech gifts measure up.

If this is the first you’ve heard about the XPRT Tech Spotlight, here’s a little background. Our hands-on testing process equips consumers with accurate information about how devices function in the real world. We test devices using our industry-standard BenchmarkXPRT tools: WebXPRT, MobileXPRT, TouchXPRT, CrXPRT, BatteryXPRT, and HDXPRT. In addition to benchmark results, we include photographs, specs, and prices for all products. New devices come online weekly, and you can browse the full list of almost 200 that we’ve featured to date on the Spotlight page.

If you represent a device vendor and want us to feature your product in the XPRT Tech Spotlight, please visit the website for more details.

Justin

The MobileXPRT 3 source code is now available

We’re excited to announce that the MobileXPRT 3 source code is now available to BenchmarkXPRT Development Community members!

Download the MobileXPRT 3 source here (login required).

We’ve also posted a download link on the MobileXPRT tab in the Members’ Area, where you will find instructions for setting up and configuring a local instance of MobileXPRT 3.

As part of our community model for software development, source code for each of the XPRTs is available to anyone who joins the community. If you’d like to review XPRT source code, but haven’t yet joined the community, we encourage you to join! Registration is quick and easy, and if you work for a company or organization with an interest in benchmarking, you can join the community for free. Simply fill out the form with your company e-mail address and select the option to be considered for a free membership. We’ll contact you to verify the address and then activate your membership.

If you have any other questions about community membership or XPRT source code, feel free to contact us. We look forward to hearing from you!

Justin

MobileXPRT 3 is here!

We’re excited to announce that MobileXPRT 3 is now available to the public! MobileXPRT 3 is the latest version of our popular tool for evaluating the performance of Android devices. The BenchmarkXPRT Development Community has been using a community preview for several weeks, but now anyone can run the tool and publish their results.

Compatible with systems running Android 5.0 and above, MobileXPRT 3 includes the same performance workloads as MobileXPRT 2015 (Apply Photo Effects, Create Photo Collages, Create Slideshow, Encrypt Personal Content, and Detect Faces to Organize Photos), plus a new optical character recognition-based workload called Scan Receipts for Spreadsheet.

MobileXPRT 3 is available at MobileXPRT.com and on the Google Play Store. Alternatively, you can download the app using either of the links below:



After trying out MobileXPRT 3, please submit your scores here and send any comments to BenchmarkXPRTsupport@principledtechnologies.com. To see test results from a variety of systems, go to MobileXPRT.com and click View Results, where you’ll find scores from a variety of Android devices. We look forward to seeing your results!

Justin

Notes from the lab: choosing a calibration system for MobileXPRT 3

Last week, we shared some details about what to expect in MobileXPRT 3. This week, we want to provide some insight into one part of the MobileXPRT development process, choosing a calibration system.

First, some background. For each of the benchmarks in the XPRT family, we select a calibration system using criteria we’ll explain below. This system serves as a reference point, and we use it to calculate scores that will help users understand a single benchmark result. The calibration system for MobileXPRT 2015 is the Motorola DROID RAZR M. We structured our calculation process so that the mean performance score from repeated MobileXPRT 2015 runs on that device is 100. A device that completes the same workloads 20 percent faster than the DROID RAZR M would have a performance score of 120, and one that performs the test 20 percent more slowly would have a score of 80. (You can find a more in-depth explanation of MobileXPRT score calculations in the Exploring MobileXPRT 2015 white paper.)

When selecting a calibration device, we are looking for a relevant reference point in today’s market. The device should be neither too slow to handle modern workloads nor so fast that it outscores most devices on the market. It should represent a level of performance that is close to what the majority of consumers experience, and one that will continue to be relevant for some time. This approach helps to build context for the meaning of the benchmark’s overall score. Without that context, testers can’t tell whether a score is fast or slow just by looking at the raw number. When compared to a well-known standard such as the calibration device, however, the score has more informative value.

To determine a suitable calibration device for MobileXPRT 3, we started by researching the most popular Android phones by market share around the world. It soon became clear that in many major markets, the Samsung Galaxy S8 ranked first or second, or at least appeared in the top five. As last year’s first Samsung flagship, the S8 is no longer on the cutting edge, but it has specs that many current mid-range phones are deploying, and the hardware should remain relevant for a couple of years.

For all of these reasons, we made the Samsung Galaxy S8 the calibration device for MobileXPRT 3. The model in our lab has a Qualcomm Snapdragon 835 SoC, 4 GB of RAM, and runs Android 7.0 (Nougat). We think it has the balance we’re looking for.

If you have any questions or concerns about MobileXPRT 3, calibration devices, or score calculations, please let us know. We look forward to sharing more information about MobileXPRT 3 as we get closer to the community preview.

Justin

Check out the other XPRTs:

Forgot your password?