PT-Logo
Forgot your password?
BenchmarkXPRT Blog banner

Category: Mobile devices

The MobileXPRT 3 Community Preview is here!

The MobileXPRT 3 Community Preview is now available to BenchmarkXPRT Development Community members! If you are not familiar with the updates and changes we implemented in the newest version of MobileXPRT, you can read more in the XPRT blog.

Members can access the MobileXPRT 3 Community Preview APK on the MobileXPRT tab in the Members’ Area. We also posted an installation guide that provides a general overview of the app and walks users through each step. The entire process takes about five minutes on most devices.

If you have any questions about MobileXPRT 3 or joining the community, please let us know. Happy testing!

Justin

The MobileXPRT 3 Community Preview is almost here!

We appreciate everyone’s patience and feedback during the MobileXPRT 3 development process. We’re in the process of wrapping up some final testing and expect to release the Community Preview (CP) within the next week.

The first thing testers will notice about the CP is a new UI/UX experience. The aesthetic is completely different than MobileXPRT 2015. We’ve made it easy to select and deselect individual workloads by tapping the workload name, and we’ve consolidated common menu items into an Android-style task bar at the bottom of the screen to improve navigation.

Fundamentally, we rebuilt MobileXPRT with Android Studio SDK 27 to bring it up to date with contemporary Android development standards. While we kept the five workloads from MobileXPRT 2015, we gave one a major overhaul, updated the test content in the remaining four, and added a sixth workload: Optical Character Recognition. You can find more details about these changes in our earlier discussion about MobileXPRT 3 development in the blog.

The screenshots below show the MobileXPRT 3 main screen, how the screen looks when users customize the combination of workloads, the end-of-test results screen, and the archived results page with one test result view expanded. Note that the archived results page also displays the median of all overall test scores completed to date.

We’ll announce the CP in a community message and here on the blog when it goes live. As with all community previews, the MobileXPRT 3 CP will be available only to BenchmarkXPRT Development Community members. If you have any questions about MobileXPRT 3 or joining the community, please let us know. Happy testing!

Justin

MobileXPRT 3 start screen   MobileXPRT 3 select workloadsMobileXPRT 3 results   MobileXPRT 3 results library

The XPRT Spotlight Black Friday Showcase helps you shop with confidence

Black Friday and Cyber Monday are almost here, and you may be feeling overwhelmed by the sea of tech gifts to choose from. The XPRTs are here to help. We’ve gathered the product specs and performance facts for some of the hottest tech devices in one convenient place—the XPRT Spotlight Black Friday Showcase. The Showcase is a free shopping tool that provides side-by-side comparisons of some of the season’s most popular smartphones, laptops, Chromebooks, tablets, and PCs. It helps you make informed buying decisions so you can shop with confidence this holiday season.

Want to know how the Google Pixel 3 stacks up against the Apple iPhone XS or Samsung Galaxy Note9 in web browsing performance or screen size? Simply select any two devices in the Showcase and click Compare. You can also search by device type if you’re interested in a specific form factor such as consoles or tablets.

The Showcase doesn’t go away after Black Friday. We’ll rename it the XPRT Holiday Buying Showcase and continue to add devices throughout the shopping season. So be sure to check back in and see how your tech gifts measure up.

If this is the first you’ve heard about the XPRT Weekly Tech Spotlight, here’s a little background. Our hands-on testing process equips consumers with accurate information about how devices function in the real world. We test devices using our industry-standard BenchmarkXPRT tools: WebXPRT, MobileXPRT, TouchXPRT, CrXPRT, BatteryXPRT, and HDXPRT. In addition to benchmark results, we include photographs, specs, and prices for all products. New devices come online weekly, and you can browse the full list of almost 150 that we’ve featured to date on the Spotlight page.

If you represent a device vendor and want us to feature your product in the XPRT Weekly Tech Spotlight, please visit the website for more details.

Do you have suggestions for the Spotlight page or device recommendations? Let us know!

Justin

A BatteryXPRT bug fix is on the way

Some time ago, we started to see unusual BatteryXPRT battery life estimates and high variance on some devices when running tests at the default length of 5.25 hours (seven 45-minute iterations). We suspected that the problem resulted from changes in how new OS versions report battery life on certain devices (e.g., charging past a reported level of 100 percent). In addition, the progress of battery technology in general means that the average phone battery lasts much longer than it did a few years ago. Together, these factors sometimes led to BatteryXPRT runs where the OS reported little to no battery decrease during the first few iterations of a test. We concluded that 5.25 hours wasn’t long enough to produce an accurate battery life estimate.

After extensive experimentation and testing, we’ve decided to release a new build that increases the default BatteryXPRT test length from 5.25 hours (seven iterations) to 45 hours (60 iterations) to allow enough time for a full rundown on most phones. Based on our testing, we consider full rundown tests to be the most accurate and will use those exclusively in our Spotlight testing and elsewhere. Testers will still have the option of choosing shorter test durations, but BatteryXPRT will flag the results with a qualifier that recommends performing a full rundown.

We plan to release the updated build by the end of next week and update BatteryXPRT documentation to reflect the changes. We have not changed any of the workloads and both performance results and full-rundown battery life estimates will be comparable to results from earlier builds.

BatteryXPRT continues to be a useful tool for gauging the performance and expected battery life of Android devices while simulating real-world tasks. If you have any questions about BatteryXPRT, please feel free to ask!

Justin

Notes from the lab: choosing a calibration system for MobileXPRT 3

Last week, we shared some details about what to expect in MobileXPRT 3. This week, we want to provide some insight into one part of the MobileXPRT development process, choosing a calibration system.

First, some background. For each of the benchmarks in the XPRT family, we select a calibration system using criteria we’ll explain below. This system serves as a reference point, and we use it to calculate scores that will help users understand a single benchmark result. The calibration system for MobileXPRT 2015 is the Motorola DROID RAZR M. We structured our calculation process so that the mean performance score from repeated MobileXPRT 2015 runs on that device is 100. A device that completes the same workloads 20 percent faster than the DROID RAZR M would have a performance score of 120, and one that performs the test 20 percent more slowly would have a score of 80. (You can find a more in-depth explanation of MobileXPRT score calculations in the Exploring MobileXPRT 2015 white paper.)

When selecting a calibration device, we are looking for a relevant reference point in today’s market. The device should be neither too slow to handle modern workloads nor so fast that it outscores most devices on the market. It should represent a level of performance that is close to what the majority of consumers experience, and one that will continue to be relevant for some time. This approach helps to build context for the meaning of the benchmark’s overall score. Without that context, testers can’t tell whether a score is fast or slow just by looking at the raw number. When compared to a well-known standard such as the calibration device, however, the score has more informative value.

To determine a suitable calibration device for MobileXPRT 3, we started by researching the most popular Android phones by market share around the world. It soon became clear that in many major markets, the Samsung Galaxy S8 ranked first or second, or at least appeared in the top five. As last year’s first Samsung flagship, the S8 is no longer on the cutting edge, but it has specs that many current mid-range phones are deploying, and the hardware should remain relevant for a couple of years.

For all of these reasons, we made the Samsung Galaxy S8 the calibration device for MobileXPRT 3. The model in our lab has a Qualcomm Snapdragon 835 SoC, 4 GB of RAM, and runs Android 7.0 (Nougat). We think it has the balance we’re looking for.

If you have any questions or concerns about MobileXPRT 3, calibration devices, or score calculations, please let us know. We look forward to sharing more information about MobileXPRT 3 as we get closer to the community preview.

Justin

News from the MobileXPRT 3 team

A few months ago, we shared some of our thoughts during the early planning stages of MobileXPRT 3 development. Since then, we’ve started building the new benchmark with Android Studio SDK 27. We’re now at a place where we can share more details about what to expect in MobileXPRT 3. In a nutshell, one of the five workloads in the previous version, MobileXPRT 2015, is getting a major overhaul, the remaining four workloads are getting updated test content, and we’re adding one completely new workload.

One of the first challenges we tackled was to completely rebuild the Create Slideshow workload. In MobileXPRT 2015, the workload uses FFmpeg to convert photos into video. FFmpeg utilizes a C++ executable, and it needs to be compiled differently for different architectures such as x86, x64, arm32, arm64, etc. With each new Android version, the task of maintaining FFmpeg compatibility with numerous architectures and Android versions becomes more complex. MobileXPRT 2015 still works well on most Android devices, but we wanted a more future-proof solution. In MobileXPRT 3, the Create Slideshow workload will use the Android MediaCodec API instead of FFmpeg. This change enables the workload to run successfully on devices that could not complete the workload in MobileXPRT 2015.

We are updating the test content of the following workloads: Apply Photo Effects, Create Photo Collages, Encrypt Personal Content, and Detect Faces to Organize Photos. We will replace items such as photos and videos with more contemporary file resolutions and sizes where applicable.

In the mobile device market, artificial intelligence and machine learning capabilities are rapidly moving from the level of novelty to being integrated into many daily tasks, so we wanted to include an AI or ML element in MobileXPRT 3. Our new workload uses Google’s Mobile Vision API to perform optical character recognition (OCR) tasks involving scanning receipts for personal records or an expense report. The scenario is similar to the OCR receipt-scanning task in WebXPRT 3, though the two workloads are based on different text-recognition technologies.

Finally, we’re updating the MobileXPRT UI to improve the look of the benchmark and make it easier to use. We’ll share a sneak peek of the new UI here in the blog around the time of the community preview. If you have any questions about MobileXPRT 2015 or MobileXPRT 3, please let us know!

Justin

Check out the other XPRTs: