We’re excited to announce that the CrXPRT 2 Community Preview (CP) is now available! BenchmarkXPRT Development Community members can access the preview using a direct link posted on the CrXPRT tab in the XPRT Members’ Area (login required), where they will also find the CrXPRT 2 CP user manual.
You can find more information about the key differences between CrXPRT 2015 and CrXPRT 2 in last week’s blog entry. During the preview period, we allow testers to publish CP test scores, but CrXPRT 2 overall performance test scores and battery life measurements are not comparable to CrXPRT 2015 scores.
If you have any questions about CrXPRT 2 or joining the community, please let us know!
We appreciate everyone’s
patience and feedback during the CrXPRT 2 development process. We’re excited to
say that we’re now wrapping up some final details and expect to release the
Community Preview (CP) within the next week.
Here is a summary of the key
differences between CrXPRT 2015 and CrXPRT 2:
As
we mentioned a few weeks ago, CrXPRT 2 has a completely
new UI in line with the functional and aesthetic themes we used for the latest
versions of WebXPRT, MobileXPRT, and HDXPRT, with a focus on intuitive
navigation.
The
CrXPRT 2 performance test includes six of the seven workloads in CrXPRT 2015.
Newer versions of Chrome can’t run the Photo Collage workload without a workaround, so we removed it from CrXPRT
2.
We
updated the images in the Photo Effects and Face Detection workloads to reflect
more contemporary file resolutions and sizes.
The
CrXPRT 2 battery life test requires a full rundown, so you’ll need charge your device
to 100 percent before you can start the test, and the length of battery life
tests will vary according to the battery life of the systems under test.
We
no longer require testers to enter luminance and audio measurements in order to
run a battery life test.
We
added a second video playback segment to each battery life iteration.
We
allow testers to publish CP test scores, but CrXPRT 2 overall performance test scores
and battery life measurements are not comparable to CrXPRT 2015 scores.
Only BenchmarkXPRT Development Community members will be able to download the CrXPRT 2 CP. Because the Chrome team stopped providing search and browse functions for hosted and packaged Chrome apps in the Chrome Web Store, members will need a direct link to access the app. Once the app is available, we’ll post that link, along with the CrXPRT 2 CP user manual, on the CrXPRT tab in the XPRT Members’ Area (login required). We’ll also send a message to the community and post a notice here in the blog.
If you have any questions about CrXPRT
2 or joining the community,
please let us know!
It’s been about two years since we released WebXPRT 3, and we’re starting to think about the WebXPRT 4 development cycle. With over 529,000 runs to date, WebXPRT continues to be our most popular benchmark because it’s quick and easy to run, it runs on almost anything with a web browser, and it evaluates performance using the types of web technologies that many people use every day.
For each new version of WebXPRT, we start the development process by looking at browser trends and analyzing the feasibility of incorporating new web technologies into our workload scenarios. For example, in WebXPRT 3, we updated the Organize Album workload to include an image-classification task that uses deep learning. We also added an optical character recognition task to the Encrypt Notes and OCR scan workload, and introduced a new Online Homework workload that combined part of the DNA Sequence Analysis scenario with a writing sample/spell check scenario.
Here are the current WebXPRT 3 workloads:
Photo Enhancement: Applies three effects, each using Canvas, to two photos.
Organize Album Using AI: Detects faces and classifies images using the ConvNetJS neural network library.
Stock Option Pricing: Calculates and displays graphic views of a stock portfolio using Canvas, SVG, and dygraphs.js.
Encrypt Notes and OCR Scan: Encrypts notes in local storage and scans a receipt using optical character recognition.
Sales Graphs: Calculates and displays multiple views of sales data using InfoVis and d3.js.
Online Homework: Performs science and English assignment tasks using Web Workers and Typo.js spell check.
What new technologies or workload scenarios should we add? Are there any existing features we should remove? Would you be interested in an associated battery life test? We want to hear your thoughts and ideas about WebXPRT, so please tell us what you think!
Last
month, Bill announced
that we were starting work on a new data center benchmark. CloudXPRT
will measure the performance of modern, cloud-first applications deployed on infrastructure
as a service (IaaS) platforms—on-premises platforms,
externally hosted platforms, and hybrid clouds that use a mix of the two. Our
ultimate goal is for CloudXPRT to use cloud-native components on an actual
stack to produce end-to-end performance metrics that can help users determine the
right IaaS configuration for their business.
Today,
we want to provide a quick update on CloudXPRT development and testing.
Installation. We’ve completely automated the CloudXPRT installation process, which leverages Kubernetes or Ansible tools depending on the target platform. The installation processes differ slightly for each platform, but testing is the same.
Workloads. We’re currently testing potential workloads that focus on three areas: web microservices, data analytics, and container scaling. We might not include all of these workloads in the first release, but we’ll keep the community informed and share more details about each workload as the picture becomes clearer. We are designing the workloads so that testers can use them to directly compare IaaS stacks and evaluate whether any given stack can meet service level agreement (SLA) thresholds.
Platforms. We want CloudXPRT to eventually support testing on a variety of popular externally hosted platforms. However, constructing a cross-platform benchmark is complicated and we haven’t yet decided which external platforms the first CloudXPRT release will support. We’ve successfully tested the current build with on-premises IaaS stacks and with one externally hosted platform, Amazon Web Services. Next, we will test the build on Google Cloud Hosting and Microsoft Azure.
Timeline. We are on track to meet our target of releasing a CloudXPRT preview build in late March and the first official build about two months later. If anything changes, we’ll post an updated timeline here in the blog.
If
you would like to share any thoughts or comments related to CloudXPRT or cloud
benchmarking, please feel free to contact
us.
Yesterday, we received a report that an HDXPRT 4 tester encountered an error message during the Convert Videos workload. During the workload, HDXPRT uses HandBrake 1.2.2 and CyberLink MediaEspresso 7.5 to convert multiple videos to formats optimized for mobile phones.
The error message reports that the video files did not load correctly:
We
apologize for the inconvenience that this causes for HDXPRT testers. We’re
troubleshooting to determine the cause of the issue and will let the community
know as soon as we identify a reliable solution. If you have any insight into
this issue, or have encountered any other error messages during HDXPRT testing,
please feel free to contact us!
Microsoft recently released a new Chromium-based version of the Edge browser, and several tech press outlets have released reviews and results from head-to-head browser performance comparison tests. Because WebXPRT is a go-to benchmark for evaluating browser performance, PCMag, PCWorld, and VentureBeat, among others, used WebXPRT 3 scores as part of the evaluation criteria for their reviews.
We thought we
would try a quick experiment of our own, so we grabbed a recent laptop from our
Spotlight testbed: a Dell XPS 13 7930 running
Windows 10 Home 1909 (18363.628) with an Intel Core i3-10110U processor and 4
GB of RAM. We tested on a clean system image after installing all current
Windows updates, and after the update process completed, we turned off updates
to prevent them from interfering with test runs. We ran WebXPRT 3 three times on
six browsers: a new browser called Brave, Google Chrome, the legacy version of
Microsoft Edge, the new version of Microsoft Edge, Mozilla Firefox, and Opera.
The posted score for each browser is the median of the three test runs.
As you can
see in the chart below, five of the browsers (legacy Edge, Brave, Opera, Chrome,
and new Edge) produced scores that were nearly identical. Mozilla Firefox was
the only browser that produced a significantly different score. The parity
among Brave, Chrome, Opera, and the new Edge is not that surprising,
considering they are all Chromium-based browsers. The rank order and relative
scaling of these results is similar to the results published by the tech
outlets mentioned above.
Do these
results mean that Mozilla Firefox will provide you with a speedier web
experience? Generally, a device with a higher WebXPRT score is probably going
to feel faster to you during daily use than one with a lower score. For
comparisons on the same system, however, the answer depends in part on the
types of things you do on the web, how the extensions you’ve installed affect
performance, how frequently the browsers issue updates and incorporate new web
technologies, and how accurately the browsers’ default installation settings reflect
how you would set up the same browsers for your daily workflow.
In addition,
browser speed can increase or decrease significantly after an update, only to
swing back in the other direction shortly thereafter. OS-specific optimizations
can also affect performance, such as with Edge on Windows 10 and Chrome on
Chrome OS. All of these variables are important to keep in mind when
considering how browser performance comparison results translate to your
everyday experience. In such a competitive market, and with so many variables
to consider, we’re happy that WebXPRT can help consumers by providing reliable,
objective results.
What are your
thoughts on today’s competitive browser market? We’d love to hear from you.
Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.
