Category: Mobile devices

News from the MobileXPRT 3 team

on September 27, 2018

A few months ago, we shared some of our thoughts during the early planning stages of MobileXPRT 3 development. Since then, we’ve started building the new benchmark with Android Studio SDK 27. We’re now at a place where we can share more details about what to expect in MobileXPRT 3. In a nutshell, one of the five workloads in the previous version, MobileXPRT 2015, is getting a major overhaul, the remaining four workloads are getting updated test content, and we’re adding one completely new workload.

One of the first challenges we tackled was to completely rebuild the Create Slideshow workload. In MobileXPRT 2015, the workload uses FFmpeg to convert photos into video. FFmpeg utilizes a C++ executable, and it needs to be compiled differently for different architectures such as x86, x64, arm32, arm64, etc. With each new Android version, the task of maintaining FFmpeg compatibility with numerous architectures and Android versions becomes more complex. MobileXPRT 2015 still works well on most Android devices, but we wanted a more future-proof solution. In MobileXPRT 3, the Create Slideshow workload will use the Android MediaCodec API instead of FFmpeg. This change enables the workload to run successfully on devices that could not complete the workload in MobileXPRT 2015.

We are updating the test content of the following workloads: Apply Photo Effects, Create Photo Collages, Encrypt Personal Content, and Detect Faces to Organize Photos. We will replace items such as photos and videos with more contemporary file resolutions and sizes where applicable.

In the mobile device market, artificial intelligence and machine learning capabilities are rapidly moving from the level of novelty to being integrated into many daily tasks, so we wanted to include an AI or ML element in MobileXPRT 3. Our new workload uses Google’s Mobile Vision API to perform optical character recognition (OCR) tasks involving scanning receipts for personal records or an expense report. The scenario is similar to the OCR receipt-scanning task in WebXPRT 3, though the two workloads are based on different text-recognition technologies.

Finally, we’re updating the MobileXPRT UI to improve the look of the benchmark and make it easier to use. We’ll share a sneak peek of the new UI here in the blog around the time of the community preview. If you have any questions about MobileXPRT 2015 or MobileXPRT 3, please let us know!

Justin

Posted in AI, Android, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Community Preview, Machine learning, Mobile devices, MobileXPRT, MobileXPRT 2015, MobileXPRT 3, Performance benchmarking |

Check out our new WebXPRT video!

By Justin Greene

on September 20, 2018

At over 305,000 runs and counting, WebXPRT is our most popular benchmark app. Device manufacturers, tech journalists, and developers around the world use WebXPRT because test runs are quick and easy, it runs on almost anything with a web browser, and it provides reliable data about how well devices perform when completing real-world tasks.

WebXPRT is not just for “techies,” however. To help explain what WebXPRT does and why it matters to everyday consumers, we’ve published a new video, What is WebXPRT and why should I care? The video explains the concepts behind some of WebXPRT’s workloads and how even small delays in common online tasks can add up to big headaches and a significant amount of wasted time. We all want to avoid those problems, and WebXPRT can help anyone that wants to see how their device, or a new device they’re thinking about buying, stacks up against the alternatives. We encourage you to check out the video below, which you can also find on YouTube and WebXPRT.com. If you have any questions about WebXPRT, please let us know!

Justin

Posted in BenchmarkXPRT, BenchmarkXPRT development community, Browser-based benchmarks, Cross-platform benchmarks, Mobile devices, Performance benchmarking, Video, Web-based testing, WebXPRT, WebXPRT 3 |

AI and the next MobileXPRT

By Justin Greene

on August 16, 2018

As we mentioned a few weeks ago, we’re in the early planning stages for the next version of MobileXPRT—MobileXPRT 3. We’re always looking for ways to make XPRT benchmark workloads more relevant to everyday users, and a new version of MobileXPRT provides a great opportunity to incorporate emerging tech such as AI into our apps. AI is everywhere and is beginning to play a huge role in our everyday lives through smarter-than-ever phones, virtual assistants, and smart homes. The challenge for us is to identify representative mobile AI workloads that have the necessary characteristics to work well in a benchmark setting. For MobileXPRT, we’re researching AI workloads that have the following characteristics:

They work offline, not in the cloud.
They don’t require additional training prior to use.
They support common use cases such as image processing, optical character recognition (OCR), etc.

We’re researching the possibility of using Google’s Mobile Vision library, but there may be other options or concerns that we’re not aware of. If you have tips for places we should look, or ideas for workloads or APIs we haven’t mentioned, please let us know. We’ll keep the community informed as we narrow down our options.

Justin

Posted in AI, Android, Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Collaborative benchmark development, Future of performance evaluation, Google, Machine learning, Mobile devices, MobileXPRT, Performance benchmarking | Tagged AI, benchmark, Google, image processing, mobile, MobileXPRT, optical character recognition |

Planning the next version of MobileXPRT

By Justin Greene

on July 26, 2018

We’re in the early planning stages for the next version of MobileXPRT, and invite you to send us any suggestions you may have. What do you like or not like about MobileXPRT? What features would you like to see in a new version?

When we begin work on a new version of any XPRT, one of the first steps we take is to assess the benchmark’s workloads to determine whether they will provide value during the years ahead. This step almost always involves updating test content such as photos and videos to more contemporary file resolutions and sizes, and it can also involve removing workloads or adding completely new scenarios. MobileXPRT currently includes five performance scenarios (Apply Photo Effects, Create Photo Collages, Create Slideshow, Encrypt Personal Content, and Detect Faces to Organize Photos). Should we stick with these five or investigate other use cases? What do you think?

As we did with WebXPRT 3 and the upcoming HDXPRT 4, we’re also planning to update the MobileXPRT UI to improve the look of the benchmark and make it easier to use.

Crucially, we’ll also build the app using the most current Android Studio SDK. Android development has changed significantly since we released MobileXPRT 2015 and apps must now conform to stricter standards that require explicit user permission for many tasks. Navigating these changes shouldn’t be too difficult, but it’s always possible that we’ll encounter unforeseen challenges at some point during the process.

Do you have suggestions for test scenarios that we should consider for MobileXPRT? Are there existing features we should remove? Are there elements of the UI that you find especially useful or have ideas for improving? Please let us know. We want to hear from you and make sure that MobileXPRT continues to meet your needs.

Justin

Posted in Android, Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Mobile devices, MobileXPRT, MobileXPRT 2015, MobileXPRT 3, Phones, Tablets | Tagged Android, benchmark, benchmarks, mobile, MobileXPRT, MobileXPRT 2015, MobileXPRT 3, phones, photos, tablets |

MWCS18 and AIXPRT: a new video

By Justin Greene

on July 12, 2018

A few weeks ago, Bill shared his first impressions from this year’s Mobile World Congress Shanghai (MWCS). “5G +” was the major theme, and there was a heavy emphasis on 5G + AI. This week, we published a video about Bill’s MWCS experience and the role that the XPRTs can play in evaluating emerging technologies such as 5G, AI, and VR. Check it out!

You can read more about AIXPRT development here. We’re still accepting responses to the AIXPRT Request for Comments, so if you would like to share your ideas on developing an AI/machine learning benchmark, please feel free to contact us.

Justin

Posted in 5G, AI, AIXPRT, AR, Augmented reality, Future of performance evaluation, Machine learning, Mobile devices, Mobile World Congress, Mobile World Congress, Performance benchmarking, Virtual reality | Tagged 5G, AI, AIXPRT, MWCS18, Shanghai, VR |

Thoughts from MWC Shanghai 2018

By Bill Catchings

on June 28, 2018

Ni hao from Shanghai! It is amazing the change that happens in a year. This year’s MWC Shanghai, like last year’s, took up about half of the Shanghai New International Expo Centre (SNIEC). “5G +” is the major theme and, unlike last year, 5G is not something in the distant future. It is now assumed to be in progress.

The biggest of the pluses was AI, with a number of booths explicitly sporting 5G + AI signage. There were also 5G plus robots, cars, and cloud services. Many of those are really about AI as well. The show makes it feel like 5G is everywhere and will make everything better (or at least a lot faster). And Asia is leading the way.

[caption id="attachment_3447" align="alignleft" width="640"]

5G + robotics at MWCS 18.[/caption]

Most of the booths touted their 5G support as they did last year, but rather than talking about the future, they tried to say that their 5G was now. They claimed their products were in real-world tests with anticipated deployment schedules. One of the keynote speakers talked about 1.2 billion 5G connections by 2025, with more than half of those in Asia. The purported scale and speed of the transition to 5G is staggering.

[caption id="attachment_3449" align="alignleft" width="640"]

The keynote stage, displaying some big numbers.[/caption]

The last two halls I visited showed that world is not all 5G and AI. These halls looked at current fun applications of mobile technologies and companies developing technologies in the near future. MWC allowed children into one of the halls, where they (and we adults) could fly drones and experience VR technology. I watched in some amusement as people crashed drones, rode bikes with VR gear to simulate horses, were 3D scanned, and generally tried out new tech that didn’t always work.

The second hall included small booths from new companies working on future technologies that might be ready “4 years from now” (4YFN). These companies did not have much to show yet, but each booth displayed the company name and a short phrase summing up their future tech. That led to “Deepscent Labs is a smart scent data company,” ChineSpain is a “Marketplace of experiences for Chinese tourists in Spain,” and “Juice is a tech-based music contents startup that creates an ecosystem of music.” The mind boggles!

The XPRTs’ foray into AI with AIXPRT seems well timed based on this show. Other areas from this show that may be worth considering for the XPRTs are 5G and the cloud. We would love to hear your thoughts on those areas. We know they are important, but do you need the XPRTs and their emphasis on real-world benchmarks and workloads in those areas? Drop us a line and let us know!

Bill

Posted in 5G, AI, AIXPRT, AR, Augmented reality, Automated cars, Future of performance evaluation, IoT, Mobile data, Mobile devices, Mobile World Congress, Virtual reality | Tagged 5G, AI, China, mobile, Mobile World Shanghai, MWC, Shanghai, VR |

Category: Mobile devices

News from the MobileXPRT 3 team

Check out our new WebXPRT video!

AI and the next MobileXPRT

Planning the next version of MobileXPRT

MWCS18 and AIXPRT: a new video

Thoughts from MWC Shanghai 2018

Check out the other XPRTs: