This week, we published the Exploring WebXPRT 4 white paper. It describes the design and structure of WebXPRT 4, including detailed information about the benchmark’s harness, HTML5 and WebAssembly (WASM) capability checks, and changes we’ve made to the structure of the performance test workloads. We explain the benchmark’s scoring methodology, how to automate tests, and how to submit results for publication. The white paper also includes information about the third-party functions and libraries that WebXPRT 4 uses during the HTML5 and WASM capability checks and performance workloads.
The Exploring WebXPRT 4 white paper promotes
the high level of transparency and disclosure that is a core value of the
BenchmarkXPRT Development Community. We’ve always believed that transparency
builds trust, and trust is essential for a healthy benchmarking community.
That’s why we involve community members in the benchmark development process
and disclose how we build our benchmarks and how they work.
new to the XPRT benchmarks may not know about one of the free resources we
offer. The XPRT results database currently holds more than 3,000 test results
from over 120 sources, including major tech review publications around the
world, OEMs, and independent testers. It offers a wealth of current and
historical performance data across all the XPRT benchmarks and hundreds of
We update the results
database several times a week, adding selected results from our own internal
lab testing, reliable tech media sources, and end-of-test user submissions.
(After you run one of the XPRTs, you can choose to submit the results, but they
don’t automatically appear in the database.) Before adding a result, we
evaluate whether the score makes sense and is consistent with general
expectations, which we can do only when we have sufficient system information details.
For that reason, we ask testers to disclose as much hardware and software
information as possible when publishing or submitting a result.
We encourage visitors to our site to explore the XPRT results database. There are three primary ways to do so. The first is by visiting the main BenchmarkXPRT results browser, which displays results entries for all of the XPRT benchmarks in chronological order (see the screenshot below). You can narrow the results by selecting a benchmark from the drop-down menu and can type values, such as vendor or the name of a tech publication, into the free-form filter field. For results we’ve produced in our lab, clicking “PT” in the Source column takes you to a page with additional disclosure information for the test system. For sources outside our lab, clicking the source name takes you to the original article or review that contains the result.
The second way to access our published results is by visiting the results page for an individual XPRT benchmark. Go the page of the benchmark that interests you, and look for the blue View Results button. Clicking it takes you to a page that displays results for only that benchmark. You can use the free-form filter on the page to filter those results, and can use the Benchmarks drop-down menu to jump to the other individual XPRT results pages.
The third way to view
information in our results database is with the WebXPRT 4 results viewer.
The viewer provides an information-packed, interactive environment in which
users can explore data from the curated set of WebXPRT 4 results we’ve
published on our site. To learn more about the viewer’s capabilities and
features, check out this blog post
We hope you’ll take
some time to browse the information in our results database. We welcome your feedback
about what you’d like to see in the future and suggestions for improvement. Our
database contains the XPRT scores that we’ve gathered, but we publish them as a
resource for you. Let us know
what you think!
Back in March, we discussed
the WebXPRT 4 results submission process and reminded readers that everyone who
runs a WebXPRT 4
test is welcome to submit scores for us to consider for publication in the WebXPRT 4 results viewer.
Unlike sites that publish every result that users submit, we publish only
results that meet our evaluation criteria. Among other things, scores must be
consistent with general expectations and must include enough detailed system
information to help us assess whether individual scores represent valid test
runs. Today, we offer a couple of tips to increase the likelihood that we will
publish your WebXPRT 4 test results.
Tip 1: Specify your system’s processor
While testers usually include
detailed information for the device, model number, operating system, and
browser version fields, we receive many submissions with little to no information
about the test system’s processor.
In the picture below, you can see an example of the level of detail that we require to consider a submission. We need the full processor name, including the manufacturer and model number (e.g., Intel Core i9-9980HK, AMD Ryzen 3 1300X, or Apple M1 Max). Note that we do not require the processor speed reported by the system.
Tip 2: Include a valid email
It is also common for submissions
to not include a valid email address. While we understand the privacy concerns related
to submitting a personal or corporate email address, we need a valid address
that we can use as a point of contact to confirm test-related information when
necessary. We don’t use those addresses for any other purposes, such as selling
them, sharing them with any third parties, or adding them to a mailing list.
We hope this information explains why we might not have published your results. We look forward to receiving your future score submissions. If you have any questions about the submission process, please let us know!
that WebXPRT 4 is live, we want to remind readers about the features of the WebXPRT 4 results viewer.
We’re excited about this new tool, which we view as an ongoing project that we
will expand and improve over time. The viewer currently has over 100 test
results, and we’re just getting started. We’ll continue to actively populate
the viewer with the latest PT-curated WebXPRT 4
results for the foreseeable future.
The screenshot below shows the tool’s default display. Each vertical bar in the graph represents the overall score of a single test result, with bars arranged from lowest to highest. To view a single result in detail, the user hovers over a bar until it turns white and a small popup window displays the basic details of the result. Once the user clicks to select the highlighted bar, the bar turns dark blue, and the dark blue banner at the bottom of the viewer displays additional details about that result.
the example above, the banner shows the overall score (227), the score’s
percentile rank (98th) among the scores in the current display, the name
of the test device, and basic hardware disclosure information. Users can click the
Run info button to see the run’s individual workload scores.
The viewer includes a drop-down menu to quickly filter results by major device type categories, and a tab that allows users to apply additional filtering options, such as browser type, processor vendor, and result source. The screenshot below shows the viewer after I used the device type drop-down filter to select only laptops.
The screenshot below shows the viewer as I use the filter tab to explore additional filter options, such browser type.
The viewer also lets users pin multiple specific runs, which is helpful for making side-by-side comparisons. The screenshot below shows the viewer after I pinned four runs and viewed them on the Pinned runs screen.
The screenshot below shows the viewer after I clicked the Compare runs button: the overall and individual workload scores of the pinned runs appear as a table.
We’re excited about the WebXPRT 4 results viewer, and we want to hear your feedback. Are there features you’d really like to see, or ways we can improve the viewer? Please let us know, and send us your latest test results!
We launched a preview of the WebXPRT 4 results viewer just before the new year, and have published over 75 results from a wide range of devices. We appreciate the results submissions we’ve received from independent testers so far, and will continue to populate the viewer with WebXPRT 4 Preview results from both our own testing and PT-curated external submissions.
you’ve run the test and have tried to submit results, you may have encountered
one or both of the following bugs, depending on the device type you’re testing:
You filled out the results submission
form, but the Submit button didn’t seem to do anything.
The test automatically downloaded
the results csv file multiple times.
We’ve identified the causes of the two bugs, and have instituted fixes. The bug fixes do not affect the benchmark’s workloads or scores. If you tested the WebXPRT 4 Preview and were frustrated by the results submission bugs, we apologize for the inconvenience, and invite you to retry submitting your results.
If you have any questions or comments about the WebXPRT 4 Preview or the results viewer, please feel free to contact us!
In October, we shared an early preview of the new results viewer tool that we’ve been developing in parallel with WebXPRT 4. The WebXPRT 4 Preview is now available to the public, and we’re excited to announce that the new results viewer is also live. We already have over 65 test results in the viewer, and in the weeks leading up to the WebXPRT 4 general release, we’ll be actively populating the viewer with the latest PT-curated WebXPRT 4 Preview results.
encourage readers to visit the blog for details
about the viewer’s features, and to take some time to explore the data.
We’re excited about this new tool, which we view as an ongoing project with
room for expansion and improvement based on user feedback.
If you have any questions or comments about the WebXPRT 4 Preview or the new results viewer, please feel free to contact us!