At over 412,000 runs and counting, WebXPRT is our most popular benchmark. From the first release in 2013, it’s been popular with device manufacturers, developers, tech journalists, and consumers because it’s easy to run, it runs on almost anything with a web browser, and it evaluates device performance using the types of web-based tasks that people are likely to encounter on a daily basis.
With each new version of WebXPRT, we analyze browser development trends to make sure the test’s underlying web technologies and workload scenarios adequately reflect the ways people are using their browsers to work and play. BenchmarkXPRT Development Community members can play an important part in that process by sending us feedback on existing tests and suggestions for new workloads to include.
For example, when we released WebXPRT 3, we updated the photo workloads with new images and a deep learning task used for image classification. We also added an optical character recognition task in the Encrypt Notes and OCR scan workload, and combined part of the DNA Sequence Analysis scenario with a writing sample/spell check scenario to simulate online homework in an all-new Online Homework workload.
Consider for a moment what an ideal future version of WebXPRT would look like for you. Are there new web technologies or workload scenarios that you would like to see? Would you be interested in an associated battery life test? Should we include experimental tests? We’re interested in what you have to say, so please feel free to contact us with your thoughts or questions.
If you’re just now learning about WebXPRT, we offer several resources to help you better understand the benchmark and its range of uses. For a general overview of why WebXPRT matters, watch our video titled What is WebXPRT and why should I care? To read more about the details of the benchmark’s development and structure, check out the Exploring WebXPRT 3 white paper. To see WebXPRT 2015 and WebXPRT 3 scores from a wide range of processors, visit the WebXPRT 3 Processor Comparison Chart.
We look forward to hearing from you!
We’re excited to see that users have successfully completed over 250,000 WebXPRT runs! From the original WebXPRT 2013 to the most recent version, WebXPRT 3, this tool has been popular with manufacturers, developers, consumers, and media outlets around the world because it’s easy to run, it runs quickly and on a wide variety of platforms, and it evaluates device performance using real-world tasks.
If you’ve run WebXPRT in any of the more than 458 cities and 64 countries from which we’ve received complete test data—including newcomers Lithuania, Luxembourg, Sweden, and Uruguay—we’re grateful for your help in reaching this milestone. Here’s to another quarter-million runs!
If you haven’t yet transitioned your browser testing to WebXPRT 3, now is a great time to give it a try! WebXPRT 3 includes updated photo workloads with new images and a deep learning task used for image classification. It also uses an optical character recognition task in the Encrypt Notes and OCR scan workload and combines part of the DNA Sequence Analysis scenario with a writing sample/spell check scenario to simulate online homework in the new Online Homework workload. Users carry out tasks like these on their browsers daily, making these workloads very effective for assessing how well a device will perform in the real world.
Happy testing to everyone, and if you have any questions about WebXPRT 3 or the XPRTs in general, feel free to ask!
Recently, a tester wrote in and asked for help determining why they were seeing different WebXPRT scores on two tablets with the same hardware configuration. The scores differed by approximately 7.5 percent. This can happen for many reasons, including different software stacks, but score variability can also result from different testing behavior and environments. While some degree of variability is natural, the question provides us with a great opportunity to talk about the basic benchmarking practices we follow in the XPRT lab, practices that contribute to the most consistent and reliable scores.
Below, we list a few basic best practices you might find useful in your testing. While they’re largely in the context of the WebXPRT focus on evaluating browser performance, several of these practices apply to other benchmarks as well.
- Test with clean images: We use an out-of-box (OOB) method for testing XPRT Spotlight devices. OOB testing means that other than initial OS and browser version updates that users are likely to run after first turning on the device, we change as little as possible before testing. We want to assess the performance that buyers are likely to see when they first purchase the device, before installing additional apps and utilities. This is the best way to provide an accurate assessment of the performance retail buyers will experience. While OOB is not appropriate for certain types of testing, the key is to not test a device that’s bogged down with programs that influence results unnecessarily.
- Turn off updates: We do our best to eliminate or minimize app and system updates after initial setup. Some vendors are making it more difficult to turn off updates completely, but you should always account for update settings.
- Get a feel for system processes: Depending on the system and the OS, quite a lot of system-level activity can be going on in the background after you turn it on. As much as possible, we like to wait for a stable baseline (idle) of system activity before kicking off a test. If we start testing immediately after booting the system, we often see higher variability in the first run before the scores start to tighten up.
- Disclosure is not just about hardware: Most people know that different browsers will produce different performance scores on the same system. However, testers aren’t always aware of shifts in performance between different versions of the same browser. While most updates don’t have a large impact on performance, a few updates have increased (or even decreased) browser performance by a significant amount. For this reason, it’s always worthwhile to record and disclose the extended browser version number for each test run. The same principle applies to any other relevant software.
- Use more than one data point: Because of natural variability, our standard practice in the XPRT lab is to publish a score that represents the median from at least three to five runs. If you run a benchmark only once, and the score differs significantly from other published scores, your result could be an outlier that you would not see again under stable testing conditions.
We hope those tips will make testing a little easier for you. If you have any questions about the XPRTs, or about benchmarking in general, feel free to ask!
This week, we posted an updated version of our “XPRTs around the world” infographic. From time to time, we like to give readers a broader view of the impact that the XPRTs are having around the world, and the infographic shows just how far and wide the XPRTs’ reach has grown.
Here are some numbers from the latest update:
- The XPRTs have been mentioned more than 7,800 times on over 2,500 unique sites.
- Those mentions include more than 6,800 articles and reviews.
- Those mentions originated in over 400 cities located in 58 countries on six continents. If you’re a tech reviewer based in Antarctica, we’re counting on you to help us make it a clean sweep!
- The BenchmarkXPRT Development Community now includes 203 members from 73 companies and organizations around the world.
In addition to the reach numbers, we’re excited that the XPRTs have now delivered more than 250,000 real-world results!
If you’re familiar with the run counter on WebXPRT.com, you may have noticed that the WebXPRT run tally is rising quickly. Starting with the original release of WebXPRT in early 2013, it took more than three and a half years for the combined run tally of WebXPRT 2013 and WebXPRT 2015 to reach 100,000. In the nine months since that happened, users have added 60,000 runs. The pace is picking up significantly!
We’re grateful for everyone who’s helped us get this far. Here’s to another quarter-million runs and downloads!
If you’re familiar with the run counter on WebXPRT.com, you may have noticed that WebXPRT recently passed a pretty significant milestone. Since we released WebXPRT 2013, users running WebXPRT 2013 and 2015 have successfully completed over 100,000 runs!
We’re thrilled about WebXPRT’s ongoing popularity, and we think that it’s due to the benchmark’s unique combination of characteristics: it’s easy to run, it runs quickly and on a wide variety of platforms, and it evaluates device performance using real-world tasks. Manufacturers, developers, consumers, and media outlets in more than 358 cities, from Aberdeen to Zevenaar, and 57 countries, from Argentina to Vietnam, have used WebXPRT’s easy-to-understand results to compare how well devices handle everyday tasks. WebXPRT has definitely earned its reputation as a “go-to” benchmark.
If you haven’t run WebXPRT yet, give it a try. The test is free and runs in almost any browser.
We’re grateful for everyone who’s helped us reach this milestone. Here’s to another 100,000 runs!
In the modern world, we’re awash in statistics and it’s interesting how they sometimes contradict each other. In this season, polls are an obvious example. They seldom agree and sometimes, as in the case of Brexit, they can be spectacularly wrong.
The real world is complicated and there are many ways to look at the data. An approach can be valid for certain cases, but less so for others. This is why it’s so important to explain your methods and calculations.
We’ve talked before about the statistics we use in the XPRTs. There are a set of white papers that describe how the tests work and how we perform the calculations. If you’re interested, a great place to start is the WebXPRT 2013 results calculation and confidence interval white paper, which describes the statistics in detail and links to a spreadsheet with a detailed example.
Any methodology can be improved, so if you have any ideas, let us know!