BenchmarkXPRT Development Community Blog

Decisions, decisions

on September 14, 2017

Back in April, we shared some of our initial ideas for a new version of WebXPRT, and work on the new benchmark is underway. Any time we begin the process of updating one of the XPRT benchmarks, one of the first decisions we face is how to improve workload content so it better reflects the types of technology average consumers use every day. Since benchmarks typically have a life cycle of two to four years, we want the benchmark to be relevant for at least the next couple of years.

For example, WebXPRT contains two photo-related workloads, Photo Effects and Organize Album. Photo Effects applies a series of effects to a set of photos, and Organize Album uses facial recognition technology to analyze a set of photos. In both cases, we want to use photos that represent the most relevant combination of image size, resolution, and data footprint possible. Ideally, the resulting image sizes and resolutions should differentiate processing speed on the latest systems, but not at the expense of being able to run reasonably on most current devices. We also have to confirm that the photos aren’t so large as to impact page load times unnecessarily.

The way this strategy works in practice is that we spend time researching hardware and operating system market share. Given that phones are the cameras that most people use, we look at them to help define photo characteristics. In 2017, the most widespread mobile OS is Android, and while reports vary depending on the metric used, the Samsung Galaxy S5 and Galaxy S7 are at or near the top of global mobile market share. For our purposes, the data tells us that choosing photo sizes and resolutions that mirror those of the Galaxy line is a good start, and a good chunk of Android users are either already using S7-generation technology, or will be shifting to new phones with that technology in the coming year. So, for the next version of WebXPRT, we’ll likely use photos that represent the real-life environment of an S7 user.

I hope that provides a brief glimpse into the strategies we use to evaluate workload content in the XPRT benchmarks. Of course, since the BenchmarkXPRT Development Community is an open development community, we’d love to hear your comments or suggestions!

Justin

Posted in Android, Benchmark metrics, Benchmarking, BenchmarkXPRT development community, Collaborative benchmark development, Performance benchmarking, WebXPRT, WebXPRT 2017, What makes a good benchmark? |

Keeping up with the latest Android news

By Justin Greene

on September 7, 2017

Ars Technica recently published a deep-dive review of Android 8.0 (Oreo) that contains several interesting tidbits about what the author called “Android’s biggest re-architecture, ever.” After reading the details, it’s hard to argue with that assessment.

The article’s thorough analysis includes a list of the changes Oreo is bringing to the UI, notification settings, locations service settings, and more. In addition to the types of updates that we usually see, a few key points stand out.

Project Treble, a complete reworking of Android’s foundational structure intended to increase the speed and efficiency of update delivery
A serious commitment to eliminating silent background services, giving users more control over their phone’s resources, and potentially enabling significant gains in battery life
Increased machine learning/neural network integration for text selection and recognition
A potential neural network API that allows third-party plugins
Android Go, a scaled-down version of Android tuned for budget phones in developing markets

There’s too much information about each of the points to discuss here, but I encourage anyone interested in Android development to check out the article. Just be warned that when they say “thorough,” they mean it, so it’s not exactly a quick read.

Right now, Oreo is available on only the Google Pixel and Pixel XL phones, and will not likely be available to most users until sometime next year. Even though widespread adoption is a way off, the sheer scale of the expected changes requires us to adopt a long-term development perspective.

We’ll continue to track developments in the Android world and keep the community informed about any impact that those changes may have on MobileXPRT and BatteryXPRT. If you have any questions or suggestions for future XPRT/Android applications, let us know!

Justin

Posted in Android, Android O, Battery life, Google, Machine learning, Mobile devices, MobileXPRT, Open Source, Phones | Tagged Android, BatteryXPRT, MobileXPRT, Oreo |

WebXPRT and user-agent strings

By Justin Greene

on August 31, 2017

After running WebXPRT in Microsoft Edge, a tester recently asked why the browser information field on the results page displayed “Chrome 52 – Edge 15.15063.” It’s a good question; why would the benchmark report Chrome 52 when Microsoft Edge is the browser under test? The answer lies in understanding user-agent strings and the way that WebXPRT gathers specific bits of information.

When browsers request a web page from a hosting server, they send an array of basic header information that allows the server to determine the client’s capabilities and the best way to provide the requested content. One of these headers, the user-agent, presents a string of tokens that provide information about the application making the request, the operating system and version, rendering engine compatibility, and browser platform details. In effect, the user-agent string is a way for a browser to tell the hosting server the full extent of its capabilities.

When WebXPRT attempts to identify a browser, it references the browser token in the user-agent string.

The process is generally straightforward, but in some cases, browsers spoof information from other browsers in their user-agent strings, which makes accurate browser detection difficult. The reasons for this are complex, but they involve web development practices and the fact that some web pages are not designed to recognize and work well with new or less-popular browsers. When we released WebXPRT 2015, Microsoft Edge was new. The Edge team wanted to make sure that as much advanced web content as possible would be available to Edge users, so they created a user-agent string that declared itself to be several different browsers at once.

I can see this in action if I check Edge’s user-agent string on my system. Currently, it reports “Mozilla/5.0 (Windows NT 10.0; Win64; x64; ServiceUI 9) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063.” So, because of the way Edge’s user-agent string is constructed, and the way WebXPRT parses that information, the browser information field on WebXPRT’s results page will report “Chrome 52 – Edge 15.15063” on my system.

To try this on your system, in Edge, select the ellipses icon in the top right-hand corner, then F12 Developer Tools. Next, select the Console tab, and run “javascript:alert(navigator.userAgent).” A popup window will display the UA string.

You can find instructions for finding the user-agent string in other browsers here: http://techdows.com/2016/07/edge-ie-chrome-firefox-user-agent-strings.html.

In the next version of WebXPRT, we’ll work to refine the way that the test parses user-agent strings, and provide more accurate system information for testers. If you have any questions or suggestions regarding this topic, let us know!

Justin

Posted in Browser-based benchmarks, Google Chrome, Microsoft Edge, WebXPRT, WebXPRT 2015 | Tagged browsers, Chrome, Microsoft Edge, WebXPRT |

Introducing the XPRT Selector

By Justin Greene

on August 24, 2017

We’re proud of all the XPRT tools, each of which serves a different purpose for the people who rely on them. But for those new to the XPRTs, we wanted a way to make it easy to tell which tool will best suit each person’s specific requirements. To that end, today we’re excited to announce the XPRT Selector, an interactive web tool that helps consumers, developers, manufacturers, and reviewers zero in on exactly which XPRT tool is the right match for their needs.

Using the XPRT Selector is easy. Simply spin the dials on the wheel to choose the categories that best describe yourself, the devices and operating systems you’re working with, and the topic that interests you. Once you’ve aligned the dials, click Get results, and the Selector will present all the free XPRT tools and resources that are available to you. Along with choosing the best tools for you, the XPRT Selector also explains the purpose and capabilities of each tool.

To see the Selector in action, check out the short video below. You can take the XPRT Selector for a spin at http://www.principledtechnologies.com/benchmarkxprt/the-xprt-selector/.

All the XPRT tools have one thing in common: They help take the guesswork out of device evaluation and comparison, making them invaluable for anyone using, making, or writing about tech products. We think the XPRT Selector is a great addition to the fold!

Justin

Posted in Benchmark metrics, Benchmarking, BenchmarkXPRT, BenchmarkXPRT development community, Interactive tools, The XPRT Selector |

The XPRT Spotlight Back-to-School Roundup

By Justin Greene

on August 17, 2017

Today, we’re pleased to announce our second annual XPRT Spotlight Back-to-School Roundup, a free shopping tool that provides side-by-side comparisons of this school year’s most popular Chromebooks, laptops, tablets, and convertibles. We designed the Roundup to help buyers choosing devices for education, such as college students picking out a laptop or school administrators deciding on the devices for a grade. The Roundup can help make those decisions easier by gathering the product and performance facts these buyers need in one convenient place.

We tested the Roundup devices in our lab using the XPRT suite of benchmark tools. In addition to benchmark results, we also provide photographs, device specs, and prices.

If you haven’t yet visited the XPRT Weekly Tech Spotlight page, check it out. Every week, the Spotlight highlights a new device, making it easier for consumers to shop for a new laptop, smartphone, tablet, or PC. Recent devices in the spotlight include the Samsung Chromebook Pro, Microsoft Surface Laptop, Microsoft Surface Pro, OnePlus 5, and Apple iPad Pro 10.5”.

Vendors interested in having their devices featured in the XPRT Weekly Tech Spotlight or next year’s Roundup can visit the website for more details.

We’re always working on ways to make the Spotlight an even more powerful tool for helping with buying decisions. If you have any ideas for the page or suggestions for devices you’d like to see, let us know!

Justin

Posted in BenchmarkXPRT, Chromebooks, Education, Mobile devices, Performance benchmarking, Tablets, XPRT Weekly Tech Spotlight |

Machine learning everywhere!

By Bill Catchings

on August 10, 2017

I usually think of machine learning as an emerging technology that will have a big impact on our lives in the not too distant future through applications like autonomous driving. Everywhere I look, however, I see areas where machine learning will affect our lives much sooner in a myriad of smaller ways.

A recent article in Wired described one such example. It told about the work some MIT and Google researchers have done using machine learning to retouch photos. I would do this by using a photo editing program to do something like adjust the color saturation of a whole photo. Instead, their algorithm applies different filters to different parts of a photo. So, faces in the foreground might get different treatment than the sunset in the background.

The researchers train the neural network using professionally retouched photos. I love the idea of a program that automatically improves the look of my less-than-professional personal photos.

What I found more exciting, however, is that the researchers could make their software efficient enough to run on a smartphone in a fraction of a second. That makes it significantly more useful.

This technology is not yet available, but it seems like something that could show up in existing photo or camera apps before long. I hope to see it soon on a smartphone in my hand!

All of that made me think about how we might incorporate such an algorithm in the XPRTs. When I started reading the article, I was thinking it might fit well in our upcoming machine-learning XPRT. By the time I finished it, however, I realized it might belong in a future version of one of the other XPRTs, like MobileXPRT. What do you think?

Bill

Posted in Google, Machine learning, MobileXPRT | Tagged Google, machine learning, MLXPRT, MobileXPRT |

Decisions, decisions

Keeping up with the latest Android news

WebXPRT and user-agent strings

Introducing the XPRT Selector

The XPRT Spotlight Back-to-School Roundup

Machine learning everywhere!

Check out the other XPRTs: