PT-Logo
Forgot your password?
BenchmarkXPRT Blog banner

Category: Benchmarking

The XPRT activity we have planned for first half of 2020

Today, we want to let readers know what to expect from the XPRTs over the next several months. Timelines and details can always change, but we’re confident that community members will see CloudXPRT Community Preview (CP), updated AIXPRT, and CrXPRT 2 releases during the first half of 2020.

CloudXPRT

Last week, Bill shared some details about our new datacenter-oriented benchmark, CloudXPRT. If you missed that post, we encourage you to check it out and learn more about the need for a new kind of cloud benchmark, and our plans for the benchmark’s structure and metrics. We’re already testing preliminary builds, and aim to release a CloudXPRT CP in late March, followed by a version for general availability roughly two months later.

AIXPRT

About a month ago, we explained how the number of moving parts in AIXPRT will necessitate a different development approach than we’ve used for other XPRTs. AIXPRT will require more frequent updating than our other benchmarks, and we anticipate releasing the second version of AIXPRT by mid-year. We’re still finalizing the details, but it’s likely to include the latest versions of ResNet-50 and SSD-MobileNet, selected SDK updates, ease-of-use improvements for the harness, and improved installation scripts. We’ll share more detailed information about the release timeline here in the blog as soon as possible.

CrXPRT 2

As we mentioned in December, we’re working on CrXPRT 2, the next version of our benchmark that evaluates the performance and battery life of Chromebooks. You can find out more about how CrXPRT works both here in the blog and at CrXPRT.com.

We’re currently testing an alpha version of CrXPRT 2. Testing is going well, but we’re tweaking a few items and refining the new UI. We should start testing a CP candidate in the next few weeks, and will have firmer information for community members about a CP release date very soon.

We’re excited about these new developments and the prospect of extending the XPRTs into new areas. If you have any questions about CloudXPRT, AIXPRT, or CrXPRT 2, please feel free to ask!

Justin

CloudXPRT is on the way

A few months ago, we wrote about the possibility of creating a datacenter XPRT. In the intervening time, we’ve discussed the idea with folks both in and outside of the XPRT Community. We’ve heard from vendors of datacenter products, hosting/cloud providers, and IT professionals that use those products and services.

The common thread that emerged was the need for a cloud benchmark that can accurately measure the performance of modern, cloud-first applications deployed on modern infrastructure as a service (IaaS) platforms, whether those platforms are on-premises, hosted elsewhere, or some combination of the two (hybrid clouds). Regardless of where clouds reside, applications are increasingly using them in latency-critical, highly available, and high-compute scenarios.

Existing datacenter benchmarks do not give a clear indication of how applications will perform on a given IaaS infrastructure, so the benchmark should use cloud-native components on the actual stacks used for on-prem and public cloud management.

We are planning to call the benchmark CloudXPRT. Our goal is for CloudXPRT to address the needs described above while also including the elements that have made the other XPRTs successful. We plan for CloudXPRT to

  • Be relevant to on-prem (datacenter), private, and public cloud deployments
  • Run on top of cloud platform software such as Kubernetes
  • Include multiple workloads that address common scenarios like web applications, AI, and media analytics
  • Support multi-tier workloads
  • Report relevant metrics including both throughput and critical latency for responsiveness-driven applications and maximum throughput for applications dependent on batch processing

CloudXPRT’s workloads will use cloud-native components on an actual stack to provide end-to-end performance metrics that allow users to choose the best IaaS configuration for their business.

We’ve been building and testing preliminary versions of CloudXPRT for the last few months. Based on the progress so far, we are shooting to have a Community Preview of CloudXPRT ready in mid- to late-March with a version for general availability ready about two months later.

Over the coming weeks, we’ll be working on getting out more information about CloudXPRT and continuing to talk with interested parties about how they can help. We’d love to hear what workflows would be of most interest to you and what you would most like to see in a datacenter/cloud benchmark. Please feel free to contact us!

Bill

CES 2020: AI in action and a “smart” future

During last year’s Consumer Electronics Show (CES), one question kept coming to mind as I walked the floor: Are we approaching the tipping point where AI truly affects most people in meaningful ways on a daily basis? I think it’s safe to say that we’ve reached that point as a result of AI integration with phones. After all, for many of us, AI improves the quality of our photography, recommends words and phrases as we text and search the web, and lets us know when to allow extra drive time because traffic is heavy.

However, for me, the most intriguing aspects of this year’s CES are the glimpses of how AI will change every area of our lives, with and without mobile devices. The show floor is jam-packed with ways to integrate AI with everything from athletic shoes to pet care to the kitchen sink. Many of these ideas are fascinating on their own, and they’re all part of a much bigger picture. The next few years will see increased AI utilization in medicine, transportation, agriculture, water and energy distribution, natural resource protection, and many more areas. Our personal smart devices will connect to smart vehicles, smart homes, smart grids, and smart cities. In the near future, CES shows won’t need AI sections because AI will be a part of everything.

At each step of this journey, people will need objective data about how well their tech can handle the demands of common AI workloads. We’re excited that AIXPRT is already becoming a go-to tool for testing inference performance on laptops, desktops, and servers. There’s much more to come with AIXPRT in 2020, along with news about XPRTs in the datacenter, so stay tuned to the blog for exciting developments in the weeks to come!

I’ll leave you with pics from three of my favorite displays at this year’s show. The first is a model of Toyota’s Woven City. Toyota announced plans to build an entire mini city on existing company land near Mount Fuji. The city will house 2,000 people and will serve as an enormous real-time lab where designers and engineers can test ubiquitous AI and sensor technology. Toyota will also design the city to be fully sustainable with the use of hydrogen fuel cells and solar panels.

The second picture shows the electric Hyundai Urban Air Mobility prototype. Hyundai is partnering with Uber on this project, and the planned vertical take-off and landing (VTOL) craft will seat five passengers plus a pilot, have a range of 60 miles, and be able to recharge in less than 10 minutes. These concepts aren’t new, but battery and material sciences technologies are progressing to the point that this one may get off the ground!

The third picture shows BrainCo’s AI Prosthetic Hand display. The hand provides amputees with new levels of dexterity compared to previous prosthetics, and it uses AI to learn from the user’s patterns of movement. The idea is that the accuracy of gestures and grips will improve over time, allowing users to accomplish tasks that are impossible with existing technology. A young man in the booth was using the hand to paint beautiful and precise Chinese calligraphy. Very cool!

Justin

AIXPRT’s unique development path

With four separate machine learning toolkits on their own development schedules, three workloads, and a wide range of possible configurations and use cases, AIXPRT has more moving parts than any of the XPRT benchmark tools to date. Because there are so many different components, and because we want AIXPRT to provide consistently relevant evaluation data in the rapidly evolving AI and machine learning spaces, we anticipate a cadence of AIXPRT updates in the future that will be more frequent than the schedules we’ve used for other XPRTs in the past. With that expectation in mind, we want to let AIXPRT testers know that when we release an AIXPRT update, they can expect minimized disruption, consideration for their testing needs, and clear communication.

Minimized disruption

Each AIXPRT toolkit (Intel OpenVINO, TensorFlow, NVIDIA TensorRT, and Apache MXNet) is on its own development schedule, and we won’t always have a lot of advance notice when new versions are on the way. Hypothetically, a new version of OpenVINO could release one month, and a new version of TensorRT just two months later. Thankfully, the modular nature of AIXPRT’s installation packages ensures that we won’t need to revise the entire AIXPRT suite every time a toolkit update goes live. Instead, we’ll update each package individually when necessary. This means that if you only test with a single AIXPRT package, updates to the other packages won’t affect your testing. For us to maintain AIXPRT’s relevance, there’s unfortunately no way to avoid all disruption, but we’ll work to keep it to a minimum.

Consideration for testers

As we move forward, when software compatibility issues force us to update an AIXPRT package, we may discover that the update has a significant effect on results. If we find that results from the new package are no longer comparable to those from previous tests, we’ll share the differences that we’re seeing in our lab. As always, we will use documentation and versioning to make sure that testers know what to expect and  that there’s no confusion about which package to use.

Clear communication

When we update any package, we’ll make sure to communicate any updates in the new build as clearly as possible. We’ll document all changes thoroughly in the package readmes, and we’ll talk through significant updates here in the blog. We’re also available to answer questions about AIXPRT and any other XPRT-related topic, so feel free to ask!

Justin

Planning for the next CrXPRT

We’re currently planning the next version of CrXPRT, our benchmark that evaluates the performance and battery life of Chromebooks. If you’re unfamiliar with CrXPRT, you can find out more about how it works both here in the blog and at CrXPRT.com. If you’ve used CrXPRT, we’d love to hear any suggestions you may have. What do you like or dislike about CrXPRT? What features do you hope to see in a new version?

When we begin work on a new version of any benchmark, one of our first steps is to determine whether the workloads will provide value during the years ahead. As technology and user behavior evolve, we update test content to be more relevant. One example is when we replace photos with ones that use more contemporary file resolutions and sizes.

Sometimes the changing tech landscape prompts us to remove entire workloads and add new ones. The Photo Collage workload in CrXPRT uses Portable Native Client (PNaCl) technology, for which the Chrome team will soon end support. CrXPRT 2015 has a workaround for this issue, but the best course of action for the next version of CrXPRT will be to remove this workload altogether.

The battery life test will also change. Earlier this year, we started to see unusual battery life estimates and high variance when running tests at CrXPRT’s default battery life test length of 3.5 hours, so we’ve been recommending that users perform full rundowns instead. In the next CrXPRT, the battery life test will require full rundowns.

We’ll also be revamping the CrXPRT UI to improve the look of the benchmark and make it easier to use, as we’ve done with the other recent XPRT releases.

We really do want to hear your ideas, and any feedback you send has a chance to shape the future of the benchmark. Let us know what you think!

Justin

The XPRT Spotlight Black Friday Showcase helps you shop with confidence

Black Friday and Cyber Monday are almost here, and you may be feeling overwhelmed by the sea of tech gifts to choose from. The XPRTs are here to help. We’ve gathered the product specs and performance facts for some of the hottest tech devices in one convenient place—the XPRT Spotlight Black Friday Showcase. The Showcase is a free shopping tool that provides side-by-side comparisons of some of the season’s most popular smartphones, laptops, Chromebooks, tablets, and PCs. It helps you make informed buying decisions so you can shop with confidence this holiday season.

Want to know how the Google Pixel 4 stacks up against the Apple iPhone 11 or Samsung Galaxy Note10 in web browsing performance or screen size? Simply select any two devices in the Showcase and click Compare. You can also search by device type if you’re interested in a specific form factor such as consoles or tablets.

The Showcase doesn’t go away after Black Friday. We’ll rename it the XPRT Holiday Showcase and continue to add devices such as the Microsoft Surface Pro X throughout the shopping season. Be sure to check back in and see how your tech gifts measure up.

If this is the first you’ve heard about the XPRT Tech Spotlight, here’s a little background. Our hands-on testing process equips consumers with accurate information about how devices function in the real world. We test devices using our industry-standard BenchmarkXPRT tools: WebXPRT, MobileXPRT, TouchXPRT, CrXPRT, BatteryXPRT, and HDXPRT. In addition to benchmark results, we include photographs, specs, and prices for all products. New devices come online weekly, and you can browse the full list of almost 200 that we’ve featured to date on the Spotlight page.

If you represent a device vendor and want us to feature your product in the XPRT Tech Spotlight, please visit the website for more details.

Justin

Check out the other XPRTs: