BenchmarkXPRT Blog banner

Category: Google

AI and the next MobileXPRT

As we mentioned a few weeks ago, we’re in the early planning stages for the next version of MobileXPRT—MobileXPRT 3. We’re always looking for ways to make XPRT benchmark workloads more relevant to everyday users, and a new version of MobileXPRT provides a great opportunity to incorporate emerging tech such as AI into our apps. AI is everywhere and is beginning to play a huge role in our everyday lives through smarter-than-ever phones, virtual assistants, and smart homes. The challenge for us is to identify representative mobile AI workloads that have the necessary characteristics to work well in a benchmark setting. For MobileXPRT, we’re researching AI workloads that have the following characteristics:

  • They work offline, not in the cloud.
  • They don’t require additional training prior to use.
  • They support common use cases such as image processing, optical character recognition (OCR), etc.


We’re researching the possibility of using Google’s Mobile Vision library, but there may be other options or concerns that we’re not aware of. If you have tips for places we should look, or ideas for workloads or APIs we haven’t mentioned, please let us know. We’ll keep the community informed as we narrow down our options.

Justin

CrXPRT helps to navigate the changing Chromebook market

Some people envision Chromebooks as low-end, plastic-shelled laptops that large organizations buy in bulk because they’re inexpensive and easy to manage. While many sub-$200 Chromebooks are still available, the platform is no longer limited to budget chipsets and little memory. Consumers can now choose systems that feature up to 16 GB of RAM, 8th generation Intel Core CPUs, and Core i7 configurations for those willing to pay around $1,600. In addition, some Chromebooks can now run Android apps, Microsoft Office mobile apps, Linux apps, and even Windows apps. While Chromebooks still depend heavily on connectivity and cloud storage, an increasing number of Chrome apps let you perform substantial productivity tasks offline. The Chrome OS landscape has changed so much that for certain use cases, the practical hardware gap between Chromebooks and traditional laptops is narrowing.

More consumers might be interested in Chromebooks than was the case a few years ago, but how they make sense of all the devices on the market? CrXPRT can help by providing objective data on Chromebook performance and battery life. Steven J. Vaughan Nichols offered a great example of the value CrXPRT can provide in his recent ZDNet article on the new Core i7-based Google Pixelbook. The Pixelbook’s CrXPRT score of 226 showed that it performs everyday tasks faster than any of the Chromebooks in our results database. When trying to decide whether it’s worth spending a few hundred or even a thousand dollars more on a new Chromebook, having the right data in hand can transform guesses into well-informed decisions.

You don’t have to be a tech journalist or even a techie to use CrXPRT. If you’d like to learn more about CrXPRT, we encourage you to read the CrXPRT feature here in the blog or visit CrXPRT.com.

Justin

Keeping up with the latest Android news

Ars Technica recently published a deep-dive review of Android 8.0 (Oreo) that contains several interesting tidbits about what the author called “Android’s biggest re-architecture, ever.” After reading the details, it’s hard to argue with that assessment.

The article’s thorough analysis includes a list of the changes Oreo is bringing to the UI, notification settings, locations service settings, and more. In addition to the types of updates that we usually see, a few key points stand out.

  • Project Treble, a complete reworking of Android’s foundational structure intended to increase the speed and efficiency of update delivery
  • A serious commitment to eliminating silent background services, giving users more control over their phone’s resources, and potentially enabling significant gains in battery life
  • Increased machine learning/neural network integration for text selection and recognition
  • A potential neural network API that allows third-party plugins
  • Android Go, a scaled-down version of Android tuned for budget phones in developing markets


There’s too much information about each of the points to discuss here, but I encourage anyone interested in Android development to check out the article. Just be warned that when they say “thorough,” they mean it, so it’s not exactly a quick read.

Right now, Oreo is available on only the Google Pixel and Pixel XL phones, and will not likely be available to most users until sometime next year. Even though widespread adoption is a way off, the sheer scale of the expected changes requires us to adopt a long-term development perspective.

We’ll continue to track developments in the Android world and keep the community informed about any impact that those changes may have on MobileXPRT and BatteryXPRT. If you have any questions or suggestions for future XPRT/Android applications, let us know!

Justin

Machine learning everywhere!

I usually think of machine learning as an emerging technology that will have a big impact on our lives in the not too distant future through applications like autonomous driving. Everywhere I look, however, I see areas where machine learning will affect our lives much sooner in a myriad of smaller ways.

A recent article in Wired described one such example. It told about the work some MIT and Google researchers have done using machine learning to retouch photos. I would do this by using a photo editing program to do something like adjust the color saturation of a whole photo. Instead, their algorithm applies different filters to different parts of a photo. So, faces in the foreground might get different treatment than the sunset in the background.

The researchers train the neural network using professionally retouched photos. I love the idea of a program that automatically improves the look of my less-than-professional personal photos.

What I found more exciting, however, is that the researchers could make their software efficient enough to run on a smartphone in a fraction of a second. That makes it significantly more useful.

This technology is not yet available, but it seems like something that could show up in existing photo or camera apps before long. I hope to see it soon on a smartphone in my hand!

All of that made me think about how we might incorporate such an algorithm in the XPRTs. When I started reading the article, I was thinking it might fit well in our upcoming machine-learning XPRT. By the time I finished it, however, I realized it might belong in a future version of one of the other XPRTs, like MobileXPRT. What do you think?

Bill

Evolve or die

Last week, Google announced that it would retire its Octane benchmark. Their announcement explains that they designed Octane to spur improvement in JavaScript performance, and while it did just that when it was first released, those improvements have plateaued in recent years. They also note that there are some operations in Octane that optimize Octane scores but do not reflect real-world scenarios. That’s unfortunate, because they, like most of us, want improvements in benchmark scores to mean improvements in end-user experience.

WebXPRT comes at the web performance issue differently. While Octane’s goal was to improve JavaScript performance, the purpose of WebXPRT is to measure performance from the end user’s perspective. By doing the types of work real people do, WebXPRT doesn’t measure only improvements in JavaScript performance; it also measures the quality of the real-world user experience. WebXPRT’s results also reflect the performance of the entire device and software stack, not just the performance of the JavaScript interpreter.

Google’s announcement reminds us that benchmarks have finite life spans, that they must constantly evolve to keep pace with changes in technology, or they will become useless. To make sure the XPRT benchmarks do just that, we are always looking at how people use their devices and developing workloads that reflect their actions. This is a core element of the XPRT philosophy.

As we mentioned last week, we’ve working on the next version of WebXPRT. If you have any thoughts about how it should evolve, let us know!

Eric

Running Android-oriented XPRTs on Chrome OS

Since last summer, we’ve been following Google’s progress in bringing Android apps and the Google Play store to Chromebooks, along with their plan to gradually phase out support for Chrome apps over the next few years. Because we currently offer apps that assess battery life and performance for Android devices (BatteryXPRT and MobileXPRT) and Chromebooks (CrXPRT), the way this situation unfolds could affect the makeup of the XPRT portfolio in the years to come.

For now, we’re experimenting to see how well the Android app/Chrome OS merger is working with the devices in our lab. One test case is the Samsung Chromebook Plus, which we featured in the XPRT Weekly Tech Spotlight a few weeks ago. Normally, we would publish only CrXPRT and WebXPRT results for a Chromebook, but installing and running MobileXPRT 2015 from the Google Play store was such a smooth and error-free process that we decided to publish the first MobileXPRT score for a device running Chrome OS.

We also tried running BatteryXPRT on the Chromebook Plus, but even though the installation was quick and easy and the test kicked off without a hitch, we could not generate a valid result. Typically, the test would complete several iterations successfully, but terminate before producing a result. We’re investigating the problem, and will keep the community up to date on what we find.

In the meantime, we continue to recommend that Chromebook testers use CrXPRT for performance and battery life assessment. While we haven’t encountered any issues running MobileXPRT 2015 on Chromebooks, CrXPRT has a proven track record.

If you have any questions about running Android-oriented XPRTs on Chrome OS, or insights that you’d like to share, please let us know.

Justin

Check out the other XPRTs:

Forgot your password?