PT-Logo
Forgot your password?
BenchmarkXPRT Blog banner

Category: HTML5

Which browser is the fastest? It’s complicated.

PCWorld recently published the results of a head-to-head browser performance comparison between Google Chrome, Microsoft Edge, Mozilla Firefox, and Opera. As we’ve noted about similar comparisons, no single browser was the fastest in every test. Browser speed sounds like a straightforward metric, but the reality is complex.

For the comparison, PCWorld used three JavaScript-centric test suites (JetStream, SunSpider, and Octane), one benchmark that simulates user actions (Speedometer), a few in-house tests of their own design, and one benchmark that simulates real-world web applications (WebXPRT). Edge came out on top in JetStream and SunSpider, Opera won in Octane and WebXPRT, and Chrome had the best results in Speedometer and PCWorld’s custom workloads.

The reason that the benchmarks rank the browsers so differently is that each one has a unique emphasis and tests a specific set of workloads and technologies. Some focus on very low-level JavaScript tasks, some test additional technologies such as HTML5, and some are designed to identify strengths or weakness by stressing devices in unusual ways. These approaches are all valid, and it’s important to understand exactly what a given score represents. Some scores reflect a very broad set of metrics, while others assess a very narrow set of tasks. Some scores help you to understand the performance you can expect from a device in your everyday life, and others measure performance in scenarios that you’re unlikely to encounter. For example, when Eric discussed a similar topic in the past, he said the tests in JetStream 1.1 provided information that “can be very useful for engineers and developers, but may not be as meaningful to the typical user.”

As we do with all the XPRTs, we designed WebXPRT to test how devices handle the types of real-world tasks consumers perform every day. While lab techs, manufacturers, and tech journalists can all glean detailed data from WebXPRT, the test’s real-world focus means that the overall score is relevant to the average consumer. Simply put, a device with a higher WebXPRT score is probably going to feel faster to you during daily use than one with a lower score. In today’s crowded tech marketplace, that piece of information provides a great deal of value to many people.

What are your thoughts on browser testing? We’d love to hear from you.

Justin

WebXPRT in 2017

Over the last few weeks, we’ve discussed the future of HDXPRT and BatteryXPRT. This week, we’re discussing what’s in store for WebXPRT in 2017.

WebXPRT is our most popular tool. Manufacturers, developers, consumers, and media outlets in more than 350 cities and 57 countries have run WebXPRT over 113,000 times to date. The benchmark runs quickly and simply in most browsers and produces easy-to-understand results that are useful for comparing web browsing performance across a wide variety of devices and browsers. People love the fact that WebXPRT runs on almost any platform that has a web browser, from PCs to phones to game consoles.

More people are using WebXPRT in more places and in more ways than ever before. It’s an unquestioned success, but we think this is a good time to make it even better by beginning work on WebXPRT 2017. Any time change comes to a popular product, there’s a risk that faithful fans will lose the features and functionality they’ve grown to love. Relevant workloads, ease of use, and extensive compatibility have always been the core components of WebXPRT’s success, so we want to reassure users that we’re committed to maintaining all of those in future versions.

Some steps in the WebXPRT 2017 process are straightforward, such as the need to reassess the existing workload lineup and update content to reflect advances in commonly used technologies. Other steps, such as introducing new workloads to test emerging browser technologies, may be tricky to implement, but could offer tremendous value in the months and years ahead.

Are there test scenarios or browser technologies you would like to see in WebXPRT 2017, or tests you think we should get rid of? Please let us know. We want to hear from you and make sure that we’re crafting a performance tool that continues to meet your needs.

Bill

More details to come

As we’ve been saying the past couple of months, we’re working on a benchmark for Chrome OS. The experimentation phase is winding down, and we are starting to shape the code into a useable benchmark. The design plan will leverage existing WebXPRT tests, of course. However, we’ve gone far beyond that. The benchmark will include video playback, 3D modeling via WebGL, and even an HTML5 game.  The test also uses Chrome OS’ native execution capability. The benchmark will actually use the Portable Native Client (PNaCl), as PNaCl is the recommended tool chain for native client. It also gives the benchmark the ability to run on more platforms.

As we mentioned before, we’re including a battery test as part of the new benchmark. So far, we haven’t found a way to remove the requirement to put the device in developer mode for the battery test.

Next week, we’ll publish a design document for the community to review. As always, the design document is based on the comments and suggestions we received combined with our own research and experimentation.

Eric

Comment on this post in the forums

Interesting questions

We’ve had a couple of interesting questions about WebXPRT this week.

The first question was about the Face detect test in WebXPRT. One person, having noticed that changing the version of Firefox affected the WebXPRT score on a particular device, asked whether the test used the JavaScript Canvas element. The answer is yes, the Face detection test does use the Canvas element. It is based on the JavaScript library by Dr. Liu Liu.

As we have discussed in the past, the software stack on a device affects the benchmark scores. WebXPRT is a HTML5 benchmark and uses elements in the HTML5 specification, such as Canvas. Browsers implement HTML in their JavaScript engines, whose performance depends on the OS and the underlying platform.  So, WebXPRT scores are influenced by the browser and OS, as well as the platform.

The second question was whether it is possible to run WebXPRT without an Internet connection. Generally speaking, the answer to that is no. WebXPRT is a hosted application, and to run the official version, you must be able to connect to the WebXPRT servers.

However, community members can download the WebXPRT source and configure local servers that will run WebXPRT, if they desire. Note: As we discussed in Sources, any published results must be from the version hosted at webxprt.com.

Thanks for the questions and keep experimenting!

Eric

Comment on this post in the forums

Ending the year with a bang!

As we promised in the blog post The newest member of the family, we made the WebXPRT 2013 community preview available this week. It has already been used in a review! The AnandTech review of the Acer Iconia W510 includes results from the WebXPRT 2013 community preview for that device and for the Microsoft Surface RT and the Apple iPad 4. The review has results from the TouchXPRT 2013 community preview for the Acer Iconia W510, Microsoft Surface RT, and the ASUS VivoTab RT as well.

Obviously, we’ve been doing some testing ourselves. Here’s a sampling of the devices on which we’ve successfully run WebXPRT:

Device Processor Operating system Browser Score Confidence interval
HP Envy 2 1.8 GHz Intel Atom Z2760 Windows 8 Internet Explorer 10.0.92 201 +/- 6
Asus VivoTab RT 1.2 GHz Tegra 3 T30L Windows RT Internet Explorer 10.0.92 160 +/- 5
Kindle Fire 1.2 GHz ARM Cortex-A9 Android OS 2.3 (customized: 6.3.1_user_4107720) Safari 5 92 +/- 2
ASUS-made Google Nexus 7 1.2 GHz Tegra 3 T30L Android 4.2 Chrome 18 201 +/- 4
Motorola DroidX phone 1 GHz TI OMAP3630-1000 Android 4.5.621 Browser version 2.3.4 26 +/- 1
iPhone 5 1.3 GHz Apple A6 iOS 6.0.2 Safari 6 168 +/- 2
iPad mini 1GHz Apple A5 iOS 6.0.2 Safari 6 110 +/- 1
iPad 4 1.4 GHz Apple A6X iOS 6.0.1 Safari 6 180 +/- 2

 

As the results above show, WebXPRT can run on a wide range of devices. We are working to get results on lots of different devices and would like your help. We’ll set up a forum thread for results that starts with these. We’ll then add additional ones we produce. Please respond in the thread with results you get.

In addition to performance results, the WebXPRT 2013 community preview also provides a report on the HTML 5 capabilities of your device. For those who want to know more about the capabilities of HTML 5, there’s more good news. The W3C community released the feature-complete spec for HTML 5 and Canvas 2D this week.

You can find an explanation of scenarios in the WebXPRT 2013 community preview, and an explanation of how it calculates its results in the WebXPRT 2013 CP1 Overview. Let us know what you think. There’s still time to help us shape the final version of both WebXPRT 2013 and TouchXPRT 2013.

Eric

Comment on this post in the forums

The newest member of the family

In his blog post TouchXPRT Web test update, Bill mentioned that we would be releasing Web-based workloads for the community to try out. Although we developed them as part of TouchXPRT, the cross-platform nature of these tests suggested to us that they should stand on their own.

After seeking community input, we have decided to make them a separate benchmark. So, we are proud to announce WebXPRT. The first community preview will be available mid-next week. WebXPRT CP1 contains four workloads: Photo Effects, Face Detect, Stocks Dashboard, and Offline Notes. Because the workloads are all HMTL5 based, they run on a wide variety of devices and operating systems—from iPad tablets to Android phones to Windows computers.

As with all community previews, we are very interested in your opinion. Tell us what you like or don’t like about the workloads. Are there other use cases you’d like to see?

Now that we have three benchmarks, the old HDXPRT-centric model of the community needs updating. Earlier this week, a message went out to the community announcing that we will be reorganizing the benchmarks under the umbrella of BenchmarkXPRT. This reorganization will touch all aspects of the community, from the Web site to Facebook, Twitter, and even the memberships themselves. We’ll be rolling out these changes over the next few weeks, and we’ll keep you informed every step of the way.

Eric

Comment on this post in the forums

Check out the other XPRTs: