BenchmarkXPRT Blog banner

Category: WebXPRT 2015

Introducing the WebXPRT 2015 Processor Comparison Chart

Today, we’re excited to announce the WebXPRT 2015 Processor Comparison Chart, a new tool that makes it easier to access hundreds of PT-curated, real-world performance scores from a wide range of devices covering everything from TVs to phones to tablets to PCs.

The chart offers a quick way to browse and compare WebXPRT 2015 results grouped by processor. Unlike benchmark-score charts that may contain results from unknown sources, PT hand-selected each of the results from internal lab testing and reliable tech media sources. If we published multiple scores for an individual processor, the score presented in the chart will be an average of those scores. Users can hover over and click individual score bars for additional information about the test results and test sources for each processor.

WebXPRT proc chart capture

We think the WebXPRT Processor Comparison Chart will be a valuable resource for folks interested in performance testing and product evaluation, but the current iteration is only the beginning. We plan to add additional capabilities on a regular basis, such a detailed filtering and enhanced viewing and navigational options. It’s also possible that we may integrate other XPRT benchmarks down the road.

Most importantly, we want the chart to be a great asset for its users, and that’s where you come in. We’d love to hear your feedback on the features and types of data you’d like to see. If you have suggestions, please let us know!


WebXPRT and user-agent strings

After running WebXPRT in Microsoft Edge, a tester recently asked why the browser information field on the results page displayed “Chrome 52 – Edge 15.15063.” It’s a good question; why would the benchmark report Chrome 52 when Microsoft Edge is the browser under test? The answer lies in understanding user-agent strings and the way that WebXPRT gathers specific bits of information.

When browsers request a web page from a hosting server, they send an array of basic header information that allows the server to determine the client’s capabilities and the best way to provide the requested content. One of these headers, the user-agent, presents a string of tokens that provide information about the application making the request, the operating system and version, rendering engine compatibility, and browser platform details. In effect, the user-agent string is a way for a browser to tell the hosting server the full extent of its capabilities.

When WebXPRT attempts to identify a browser, it references the browser token in the user-agent string.

The process is generally straightforward, but in some cases, browsers spoof information from other browsers in their user-agent strings, which makes accurate browser detection difficult. The reasons for this are complex, but they involve web development practices and the fact that some web pages are not designed to recognize and work well with new or less-popular browsers. When we released WebXPRT 2015, Microsoft Edge was new. The Edge team wanted to make sure that as much advanced web content as possible would be available to Edge users, so they created a user-agent string that declared itself to be several different browsers at once.

I can see this in action if I check Edge’s user-agent string on my system. Currently, it reports “Mozilla/5.0 (Windows NT 10.0; Win64; x64; ServiceUI 9) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063.” So, because of the way Edge’s user-agent string is constructed, and the way WebXPRT parses that information, the browser information field on WebXPRT’s results page will report “Chrome 52 – Edge 15.15063” on my system.

To try this on your system, in Edge, select the ellipses icon in the top right-hand corner, then F12 Developer Tools. Next, select the Console tab, and run “javascript:alert(navigator.userAgent).” A popup window will display the UA string.

You can find instructions for finding the user-agent string in other browsers here:

In the next version of WebXPRT, we’ll work to refine the way that the test parses user-agent strings, and provide more accurate system information for testers. If you have any questions or suggestions regarding this topic, let us know!


Best practices

Recently, a tester wrote in and asked for help determining why they were seeing different WebXPRT scores on two tablets with the same hardware configuration. The scores differed by approximately 7.5 percent. This can happen for many reasons, including different software stacks, but score variability can also result from different testing behavior and environments. While some degree of variability is natural, the question provides us with a great opportunity to talk about the basic benchmarking practices we follow in the XPRT lab, practices that contribute to the most consistent and reliable scores.

Below, we list a few basic best practices you might find useful in your testing. While they’re largely in the context of the WebXPRT focus on evaluating browser performance, several of these practices apply to other benchmarks as well.

  • Test with clean images: We use an out-of-box (OOB) method for testing XPRT Spotlight devices. OOB testing means that other than initial OS and browser version updates that users are likely to run after first turning on the device, we change as little as possible before testing. We want to assess the performance that buyers are likely to see when they first purchase the device, before installing additional apps and utilities. This is the best way to provide an accurate assessment of the performance retail buyers will experience. While OOB is not appropriate for certain types of testing, the key is to not test a device that’s bogged down with programs that influence results unnecessarily.
  • Turn off updates: We do our best to eliminate or minimize app and system updates after initial setup. Some vendors are making it more difficult to turn off updates completely, but you should always account for update settings.
  • Get a feel for system processes: Depending on the system and the OS, quite a lot of system-level activity can be going on in the background after you turn it on. As much as possible, we like to wait for a stable baseline (idle) of system activity before kicking off a test. If we start testing immediately after booting the system, we often see higher variability in the first run before the scores start to tighten up.
  • Disclosure is not just about hardware: Most people know that different browsers will produce different performance scores on the same system. However, testers aren’t always aware of shifts in performance between different versions of the same browser. While most updates don’t have a large impact on performance, a few updates have increased (or even decreased) browser performance by a significant amount. For this reason, it’s always worthwhile to record and disclose the extended browser version number for each test run. The same principle applies to any other relevant software.
  • Use more than one data point: Because of natural variability, our standard practice in the XPRT lab is to publish a score that represents the median from at least three to five runs. If you run a benchmark only once, and the score differs significantly from other published scores, your result could be an outlier that you would not see again under stable testing conditions.

We hope those tips will make testing a little easier for you. If you have any questions about the XPRTs, or about benchmarking in general, feel free to ask!


XPRTs around the world

This week, we posted an updated version of our “XPRTs around the world” infographic. From time to time, we like to give readers a broader view of the impact that the XPRTs are having around the world, and the infographic shows just how far and wide the XPRTs’ reach has grown.

Here are some numbers from the latest update:

  • The XPRTs have been mentioned more than 7,800 times on over 2,500 unique sites.
  • Those mentions include more than 6,800 articles and reviews.
  • Those mentions originated in over 400 cities located in 58 countries on six continents. If you’re a tech reviewer based in Antarctica, we’re counting on you to help us make it a clean sweep!
  • The BenchmarkXPRT Development Community now includes 203 members from 73 companies and organizations around the world.

In addition to the reach numbers, we’re excited that the XPRTs have now delivered more than 250,000 real-world results!

If you’re familiar with the run counter on, you may have noticed that the WebXPRT run tally is rising quickly. Starting with the original release of WebXPRT in early 2013, it took more than three and a half years for the combined run tally of WebXPRT 2013 and WebXPRT 2015 to reach 100,000. In the nine months since that happened, users have added 60,000 runs. The pace is picking up significantly!

We’re grateful for everyone who’s helped us get this far. Here’s to another quarter-million runs and downloads!


Digging deeper

From time to time, we like to revisit the fundamentals of the XPRT approach to benchmark development. Today, we’re discussing the need for testers and benchmark developers to consider the multiple factors that influence benchmark results. For every device we test, all of its hardware and software components have the potential to affect performance, and changing the configuration of those components can significantly change results.

For example, we frequently see significant performance differences between different browsers on the same system. In our recent recap of the XPRT Weekly Tech Spotlight’s first year, we highlighted an example of how testing the same device with the same benchmark can produce different results, depending on the software stack under test. In that instance, the Alienware Steam Machine entry included a WebXPRT 2015 score for each of the two browsers that consumers were likely to use. The first score (356) represented the SteamOS browser app in the SteamOS environment, and the second (441) represented the Iceweasel browser (a Firefox variant) in the Linux-based desktop environment. Including only the first score would have given readers an incomplete picture of the Steam Machine’s web-browsing capabilities, so we thought it was important to include both.

We also see performance differences between different versions of the same browser, a fact especially relevant to those who use frequently updated browsers, such as Chrome. Even benchmarks that measure the same general area of performance, for example, web browsing, are usually testing very different things.

OS updates can also have an impact on performance. Consumers might base a purchase on performance or battery life scores and end up with a device that behaves much differently when updated to a new version of Android or iOS, for example.

Other important factors in the software stack include pre-installed software, commonly referred to as bloatware, and the proliferation of apps that sap performance and battery life.

This is a much larger topic than we can cover in the blog. Let the examples we’ve mentioned remind you to think critically about, and dig deeper into, benchmark results. If we see published XPRT scores that differ significantly from our own results, our first question is always “What’s different between the two devices?” Most of the time, the answer becomes clear as we compare hardware and software from top to bottom.


Tracking device evolution with WebXPRT ‘15

The XPRT Spotlight on the Apple iPhone 7 Plus gives us a great opportunity to look at the progression of WebXPRT 2015 scores for the iPhone line and see how hardware and software advances are often reflected in benchmark scores over time. This week, we’ll see how the evolution of Apple’s mobile CPU architecture has boosted web-browsing performance. In a future post, we’ll see the impact of iOS development.

As we’ve discussed in the past, multiple factors can influence benchmark results. While we’re currently using the iPhone as a test case, the same principles apply to all types of devices. We should also note that WebXPRT is an excellent gauge of expected web-browsing performance during real-world tasks, which is different than pure CPU performance in isolation.

When looking at WebXPRT ’15 scores in our database, we see that iPhone web-browsing performance has more than doubled in the last three years. In 2013, an iPhone 5s with an Apple A7 chip earned an overall WebXPRT ’15 score of 100. Today, a new iPhone 7 Plus with an A10 Fusion chip reports a score somewhere close to 210. The chart below shows the WebXPRT ’15 scores for a range of iPhones, with each iPhone’s CPU noted in parentheses.

Oct 20 iPhone chart

Moving forward from the A7 chip in the iPhone 5s to the A8 chip in the iPhone 6 and the A9 chip in the iPhone 6s and SE, we see consistent score increases. The biggest jump, at over 48%, appears in the transition from the A8 to the A9 chip, implying that folks upgrading from an iPhone 6 or 6 Plus to anything newer would notice a huge difference in web performance.

In general, folks upgrading from an A9-based phone (6S, 6S Plus, or SE) to an A10-based phone (7 and 7 Plus) could expect an increase in web performance of over 6.5%.

The scores we list represent the median of a range of scores for each device in our database. These scores come from our own testing, as well as from device reviews from media outlets such as AnandTech, Notebookcheck, and Tom’s Hardware. It’s worth noting that the highest A9 score in our database (AnandTech’s iPhone SE score of 205) overlaps with the lowest A10 Fusion score (Tom’s Hardware of Germany’s iPhone 7 score of 203), so while the improvement in median scores is clear, performance will vary according to individual phones and other factors.

Soon, we’ll revisit our iPhone test case to see how software updates can boost device performance without any changes in hardware. For more details on the newest iPhones, visit the Spotlight comparison page to see how iPhone 7 and 7 Plus specs and WebXPRT scores stack up.


Check out the other XPRTs:

Forgot your password?