Category: MobileXPRT 2015

The MobileXPRT 3 Community Preview is almost here!

on December 13, 2018

We appreciate everyone’s patience and feedback during the MobileXPRT 3 development process. We’re in the process of wrapping up some final testing and expect to release the Community Preview (CP) within the next week.

The first thing testers will notice about the CP is a new UI/UX experience. The aesthetic is completely different than MobileXPRT 2015. We’ve made it easy to select and deselect individual workloads by tapping the workload name, and we’ve consolidated common menu items into an Android-style task bar at the bottom of the screen to improve navigation.

Fundamentally, we rebuilt MobileXPRT with Android Studio SDK 27 to bring it up to date with contemporary Android development standards. While we kept the five workloads from MobileXPRT 2015, we gave one a major overhaul, updated the test content in the remaining four, and added a sixth workload: Optical Character Recognition. You can find more details about these changes in our earlier discussion about MobileXPRT 3 development in the blog.

The screenshots below show the MobileXPRT 3 main screen, how the screen looks when users customize the combination of workloads, the end-of-test results screen, and the archived results page with one test result view expanded. Note that the archived results page also displays the median of all overall test scores completed to date.

We’ll announce the CP in a community message and here on the blog when it goes live. As with all community previews, the MobileXPRT 3 CP will be available only to BenchmarkXPRT Development Community members. If you have any questions about MobileXPRT 3 or joining the community, please let us know. Happy testing!

Justin

Posted in Android, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Mobile devices, MobileXPRT, MobileXPRT 2015, MobileXPRT 3 |

Updates on HDXPRT 4 and MobileXPRT 3

By Justin Greene

on October 11, 2018

There’s a lot going on with the XPRTs, so we want to offer a quick update.

On the HDXPRT 4 front, we’re currently testing community preview candidate builds across a variety of laptops and desktops. Testing is going well, but as is often the case prior to a release, we’re still tweaking the code as necessary when we run into bugs. We’re excited about HDXPRT 4 and look forward to the community seeing how much faster and easier to use it is than previous versions. You can read more about what’s to come in HDXPRT 4 here.

On the MobileXPRT 3 front, proof-of-concept testing for the new and updated workloads went well, and we’re now working to implement the new UI. Below, you can see a mockup of the new MobileXPRT 3 start screen for phones. The aesthetic is completely different than MobileXPRT 2015, and is in line with the clean, bright look we used for WebXPRT 3 and HDXPRT 4. We’ve made it easy to select and deselect individual workloads by tapping the workload name (deselected workloads are grayed out), and we’ve consolidated common menu items into an Android-style taskbar at the bottom of the screen. Please note that this is an early view and some aspects of the screen will change. For instance, we’re certain that the final receipt-scanning workload won’t be called “Optical character recognition.”

We’ll share more information about HDXPRT 4 and MobileXPRT 3 in the coming weeks. If you have any questions about HDXPRT or MobileXPRT, or would like to share your ideas, please get in touch!

Justin

Posted in Android, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, HDXPRT, HDXPRT 4, HDXPRT development process, HDXPRT release cycle, MobileXPRT, MobileXPRT 2015, MobileXPRT 3, Performance benchmarking, WebXPRT, WebXPRT 3 | Tagged Android, Community, HDXPRT, laptops, mobile devices, MobileXPRT, phones |

News from the MobileXPRT 3 team

By Justin Greene

on September 27, 2018

A few months ago, we shared some of our thoughts during the early planning stages of MobileXPRT 3 development. Since then, we’ve started building the new benchmark with Android Studio SDK 27. We’re now at a place where we can share more details about what to expect in MobileXPRT 3. In a nutshell, one of the five workloads in the previous version, MobileXPRT 2015, is getting a major overhaul, the remaining four workloads are getting updated test content, and we’re adding one completely new workload.

One of the first challenges we tackled was to completely rebuild the Create Slideshow workload. In MobileXPRT 2015, the workload uses FFmpeg to convert photos into video. FFmpeg utilizes a C++ executable, and it needs to be compiled differently for different architectures such as x86, x64, arm32, arm64, etc. With each new Android version, the task of maintaining FFmpeg compatibility with numerous architectures and Android versions becomes more complex. MobileXPRT 2015 still works well on most Android devices, but we wanted a more future-proof solution. In MobileXPRT 3, the Create Slideshow workload will use the Android MediaCodec API instead of FFmpeg. This change enables the workload to run successfully on devices that could not complete the workload in MobileXPRT 2015.

We are updating the test content of the following workloads: Apply Photo Effects, Create Photo Collages, Encrypt Personal Content, and Detect Faces to Organize Photos. We will replace items such as photos and videos with more contemporary file resolutions and sizes where applicable.

In the mobile device market, artificial intelligence and machine learning capabilities are rapidly moving from the level of novelty to being integrated into many daily tasks, so we wanted to include an AI or ML element in MobileXPRT 3. Our new workload uses Google’s Mobile Vision API to perform optical character recognition (OCR) tasks involving scanning receipts for personal records or an expense report. The scenario is similar to the OCR receipt-scanning task in WebXPRT 3, though the two workloads are based on different text-recognition technologies.

Finally, we’re updating the MobileXPRT UI to improve the look of the benchmark and make it easier to use. We’ll share a sneak peek of the new UI here in the blog around the time of the community preview. If you have any questions about MobileXPRT 2015 or MobileXPRT 3, please let us know!

Justin

Posted in AI, Android, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Machine learning, Mobile devices, MobileXPRT, MobileXPRT 2015, MobileXPRT 3, Performance benchmarking |

Planning the next version of MobileXPRT

By Justin Greene

on July 26, 2018

We’re in the early planning stages for the next version of MobileXPRT, and invite you to send us any suggestions you may have. What do you like or not like about MobileXPRT? What features would you like to see in a new version?

When we begin work on a new version of any XPRT, one of the first steps we take is to assess the benchmark’s workloads to determine whether they will provide value during the years ahead. This step almost always involves updating test content such as photos and videos to more contemporary file resolutions and sizes, and it can also involve removing workloads or adding completely new scenarios. MobileXPRT currently includes five performance scenarios (Apply Photo Effects, Create Photo Collages, Create Slideshow, Encrypt Personal Content, and Detect Faces to Organize Photos). Should we stick with these five or investigate other use cases? What do you think?

As we did with WebXPRT 3 and the upcoming HDXPRT 4, we’re also planning to update the MobileXPRT UI to improve the look of the benchmark and make it easier to use.

Crucially, we’ll also build the app using the most current Android Studio SDK. Android development has changed significantly since we released MobileXPRT 2015 and apps must now conform to stricter standards that require explicit user permission for many tasks. Navigating these changes shouldn’t be too difficult, but it’s always possible that we’ll encounter unforeseen challenges at some point during the process.

Do you have suggestions for test scenarios that we should consider for MobileXPRT? Are there existing features we should remove? Are there elements of the UI that you find especially useful or have ideas for improving? Please let us know. We want to hear from you and make sure that MobileXPRT continues to meet your needs.

Justin

Posted in Android, Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Mobile devices, MobileXPRT, MobileXPRT 2015, MobileXPRT 3, Phones, Tablets | Tagged Android, benchmark, benchmarks, mobile, MobileXPRT, MobileXPRT 2015, MobileXPRT 3, phones, photos, tablets |

MobileXPRT: evaluate the performance of your Android device

By Justin Greene

on July 27, 2017

We recently discussed the capabilities and benefits of TouchXPRT, CrXPRT, BatteryXPRT, and HDXPRT. This week, we’re focusing on MobileXPRT, an app that evaluates how well an Android device handles everyday tasks. Like the other XPRT family benchmarks, MobileXPRT is easy to use. It takes less than 15 minutes to run on most devices, runs relatable workloads, and delivers reliable, objective, and easy-to-understand results.

MobileXPRT includes five performance scenarios (Apply Photo Effects, Create Photo Collages, Create Slideshow, Encrypt Personal Content, and Detect Faces to Organize Photos). By default, the benchmark runs all five tasks and reports individual workload scores and an overall performance score.

MobileXPRT 2015 is the latest version of the app, supporting both 32-bit and 64-bit hardware running Android 4.4 or higher. To test systems running older versions of Android, or to test 32-bit performance on a 64-bit system, you can use MobileXPRT 2013. The results of the two versions are comparable.

MobileXPRT is a useful tool for anyone who wants to compare the performance capabilities of Android phones or tablets. To see test results from a variety of systems, go to MobileXPRT.com and click View Results, where you’ll find scores from many different Android devices.

If you’d like to run MobileXPRT:

Simply download MobileXPRT from MobileXPRT.com or the Google Play Store. The full installer package on MobileXPRT.com, containing both app and test data, is 243 MB. You may also use this link to download the 18 MB MobileXPRT app file, which will download the test data during installation. The MobileXPRT user manual provides instructions for configuring your device and kicking off a test.

If you’d like to dig into the details:

Check out the Exploring MobileXPRT 2015 white paper. In it, we discuss the MobileXPRT development process and details of the individual performance scenarios. We also explain exactly how the benchmark calculates results.

If you’d like to dig even deeper, the MobileXPRT source code is available to members of the BenchmarkXPRT Development Community, so consider joining today. Membership is free for members of any company or organization with an interest in benchmarks, and there are no obligations after joining.

If you haven’t used MobileXPRT before, give it a shot and let us know what you think!

Justin

Posted in Android, BatteryXPRT 2014 for Android, CrXPRT, HDXPRT, Mobile devices, MobileXPRT, MobileXPRT 2015, Performance benchmarking, TouchXPRT, WebXPRT |

Notes from the lab

By Justin Greene

on June 15, 2017

This week’s XPRT Weekly Tech Spotlight featured the Alcatel A30 Android phone. We chose the A30, an Amazon exclusive, because it’s a budget phone running Android 7.0 (Nougat) right out of the box. That may be an appealing combination for consumers, but running a newer OS on inexpensive hardware such as what’s found in the A30 can cause issues for app developers, and the XPRTs are no exception.

Spotlight fans may have noticed that we didn’t post a MobileXPRT 2015 or BatteryXPRT 2014 score for the A30. In both cases, the benchmark did not produce an overall score because of a problem that occurs during the Create Slideshow workload. The issue deals with text relocation and significant changes in the Android development environment.

As of Android 5.0, on 64-bit devices, the OS doesn’t allow native code executables to perform text relocation. Instead, it is necessary to compile the executables using position-independent code (PIC) flags. This is how we compiled the current version of MobileXPRT, and it’s why we updated BatteryXPRT earlier this year to maintain compatibility with the most recent versions of Android.

However, the same approach doesn’t work for SoCs built with older 32-bit ARMv7-A architectures, such as the A30’s Qualcomm Snapdragon 210, so testers may encounter this issue on other devices with low-end hardware.

Testers who run into this problem can still use MobileXPRT 2015 to generate individual workload scores for the Apply Photo Effects, Create Photo Collages, Encrypt Personal Content, and Detect Faces workloads. Also, BatteryXPRT will produce an estimated battery life for the device, but since it won’t produce a performance score, we ask that testers use those numbers for informational purposes and not publication.

If you have any questions or have encountered additional issues, please let us know!

Justin

Posted in Android, Android N, Battery life, BatteryXPRT 2014 for Android, Benchmarking computing devices, MobileXPRT, MobileXPRT 2015 |

Category: MobileXPRT 2015

The MobileXPRT 3 Community Preview is almost here!

Updates on HDXPRT 4 and MobileXPRT 3

News from the MobileXPRT 3 team

Planning the next version of MobileXPRT

MobileXPRT: evaluate the performance of your Android device

If you’d like to run MobileXPRT:

If you’d like to dig into the details:

Notes from the lab

Check out the other XPRTs: