AI/ML testing: LLM testing with PTChatterly

PTChatterly is a new LLM benchmark and sizing framework, exclusively available from PT, that quantifies the performance and user experience your customers can expect for a solution running an in-house LLM.

Utilizing an existing LLM and the retrieval augmented generation (RAG) method, it searches a local corpus of data and constructs responses in AI-assisted chatbot conversations with multiple simulated users. It generates meaningful, real-world metrics, e.g., “32 people can have simultaneous conversations with at worst XX response time.”

For some platforms, we may need to incorporate code development/optimization time in the scope of the project; contact us to learn more.

AI/ML testing: Speech recognition

Gauge a solution’s performance for AI/ML speech recognition

An automatic speech recognition (ASR) AI/ML model does exactly what you’d expect: predict text based on audio input of one or more people speaking.¹ ASR applications are useful for captioning videos, transcribing podcasts and meetings, improving accessibility, and much more.

There are as many different ASR models as there are ways to assess their performance. To take just one example, we can test a solution’s training or inference performance on the Recurrent Neural Network Transducer (RNN-T) model using the MLperf RNN-T benchmark. MLperf’s RNN-T benchmark outputs several key metrics, including throughput in number of training sequences processed per second, latency (i.e., the time a solution takes to process a single sequence or inference), Word Error Rate (WER), Character Error Rate (CER), and training time (i.e., the time a solution takes to train the RNN-T model to convergence).

For some platforms, we may need to incorporate code development/optimization time in the scope of the project; contact us to learn more.

1. MLPerf Training v1.0 Results, accessed November 10, 2023, https://mlcommons.org/2021/06/mlperf-training-v1-0-results/.

I/O testing with Iometer

Understand how a system performs under differing loads

Iometer generates input/output (I/O) operations to stress a solution’s storage capabilities. By configuring block size and read/write ratio, we can use this synthetic workload tool to imitate common workload types such as online transaction processing (OLTP) or data analysis. It yields outputs of input/output operations per second, latency, and throughput. Alternatively, we can use Iometer to drive disk load to stress a solution while also running a different workload or benchmark for additional performance testing.

DISKSPD

Evaluate synthetic storage performance

DISKSPD is a load generator from Microsoft that allows us to create simple synthetic storage workloads containing custom mixes of reads and writes. It is useful for evaluating raw storage performance.¹ When we use it in conjunction with a benchmark such as Perfmon or VMFleet, the tool allows us to measure the number of input/output operations per second (IOPS) that a solution can sustain.

1. Microsoft, “Use DISKSPD to test workload storage performance,” accessed January 3, 2023, https://learn.microsoft.com/en-us/azure-stack/hci/manage/diskspd-overview.

Vdbench and HCIBench

Gauge storage performance with flexible synthetic workloads

Vdbench is an open-source, synthetic benchmark tool that generates disk input/output (I/O) workloads to test a solution’s storage performance, while HCIBench—short for hyper-converged infrastructure (HCI) benchmark—draws on Vdbench and the I/O generator Fio to automate testing on HCI solutions. Both tools allow us to customize the read/write ratio of an I/O workload to simulate a specific real-world workload. The benchmarks yield results in input/output operations per second (IOPS), latency, and throughput.

Backup and restore studies (enterprise)

Measure speed and features of backup and recovery solutions

For both backup and recovery, speed is of the utmost importance: A fast solution can minimize downtime, while a slow one can cause significant problems in the event of a storage loss. We can perform backups, simulate losses, and initiate recoveries at scale to test the real-world capabilities of your and/or a competitor’s backup and recovery solution. We work with you to customize what data we’re backing up, the size and type of that data, whether we perform incremental or complete backups, and more.

Deployment studies (enterprise)

Quantify ease of deployment

Deployment is the first step in putting a solution to work. We can deploy a technology ourselves, using the same steps a real-world IT team would, and generate a deployment guide or a test report quantifying the time and effort the deployment required. We can also test your and/or a competitor’s deployment services, acting as a customer and tracking the elapsed time and customer experience from first sales contact to functioning solution.

Manageability studies

Assess time and effort savings from manageability tools

Every piece of technology requires management, so organizations win when IT staff can minimize the time and effort they spend on everyday management tasks. We test manageability tools the way real IT teams use them. In addition to verifying the functionality of key features, we perform common, real-world tasks and measure how much time and how many steps they take, quantifying a tool’s ease of use. In this way, we can compare one tool to another or measure how much time companies can save by using the tool rather than a manual approach.

iPerf

Measure network speed

iPerf is a synthetic, cross-platform network test that measures maximum network bandwidth. It allows us to test with multiple network protocols: transmission control protocol (TCP), stream control transmission protocol (SCTP), and user datagram protocol (UDP). iPerf generates network protocol streams and then measures the bandwidth available, packet rate, packet loss, and more. Its output may include throughput in Gbps and packet rate in packets per second, among other data.

Thin client testing

Assess VDI performance from an end-user perspective

If your end-users log into VDI desktops every day, you may see an opportunity to cut costs by providing them with thin clients rather than full-featured laptops or desktops. The ideal thin client would be inexpensive but easy to manage and deliver a responsive experience in the apps your teams use daily. We sometimes test thin clients in the same ways we test all end-user devices, including with a variety of benchmarks and by hand-timing real-world tasks. We can also assess thin clients from other perspectives more specific to this type of device, including ease of manageability and the number of different VMs a single device can support.

JetStream

Measure web browsing performance

As a JavaScript and WebAssembly benchmark suite, JetStream runs several subtests—including “a variety of advanced workloads and programming techniques”—and combines the results into a single score.¹ It gives a higher score to browsers that start up or execute code more quickly, which could translate to smoother user experiences.

1. JetStream 2 In-Depth Analysis,” accessed October 5, 2023, https://browserbench.org/JetStream2.0/in-depth.html.

Frametest

Measure the rate of video and image transmission

Working with video requires moving large volumes of data from one location to another. Frametest is a synthetic benchmark utility that can evaluate storage performance specifically for video editing use cases. It outputs results in throughput and the number of frames per second (FPS) a solution can sustain, providing insight into such areas as video quality (a solution that handles more frames per second typically displays information more clearly); the experience of video content creators (being able to send data over a network quickly can boost productivity); the efficiency of applications that rely on transferring video, such as security and targeted advertising in retail; and storage performance.

SLOB

Assess Oracle performance

SLOB, or Silly Little Oracle Benchmark, is a tool that generates random read and write I/O operations for Oracle databases, allowing us to measure how many input/output operations per second (IOPS) a solution can handle. The ability to process more IOPS while still supporting high throughput indicates a solution’s ability to support periods of heavy user database activity. SLOB yields results in IOPS, latency, and throughput.

Spark LDA

Measure big data classification performance with Spark

The Spark LDA workload, part of the Intel HiBench suite of benchmarking software, is a topic model that “infers topics from a collection of documents.”¹ Latent Dirichlet allocation (LDA) is a real-world technique to dynamically analyze text, identifying, categorizing, and refining topics in a text document as well as summarizing the document. Businesses might use LDA to organize customer reviews on a product or recommend new products based on user history. In using this workload for performance testing, we measure throughput while the workload is running and how long it takes a solution to complete the workload.

1. "HiBench,” accessed November 8, 2023, https://github.com/Intel-bigdata/HiBench.

Spark LR

Quantify decision-making performance with Logistic Regression

A machine learning algorithm that uses a decision tree to predict a response from a set of categories and sorts data into similar groups, Logistic Regression (LR) can help deliver insights to decisionmakers or recommend products to online shoppers. The Spark LR workload, part of the Intel HiBench suite of benchmarking software, measures how well a system performs LR clustering on categorical data (data that one can categorize into groups), continuous data (data that can have any value), and binary data (data that can only take one of two values).¹ When we test with Spark LR, we can measure throughput while the workload runs and how long a solution needs to complete the workload.

1. "HiBench,” accessed November 8, 2023, https://github.com/Intel-bigdata/HiBench.

Weathervane

Quantify the performance of on-premises and cloud-based Kubernetes clusters

Weathervane is an application-level Kubernetes benchmark that, per VMware, "measures the performance capabilities of a Kubernetes cluster by deploying one or more instances of a benchmark application on the cluster and then driving a load against those applications."¹ The benchmark application in question is a multi-tier, real-time auction web app, where simulated users view items and make bids within a set time frame. Weathervane outputs results in WvUsers, representing the greatest number of simulated users the application instances can support while meeting a certain level of performance.

1. Harold Rosenberg, “Weathervane 2.0: An Application-Level Performance Benchmark for Kubernetes,” accessed January 13, 2022, https://blogs.vmware.com/performance/2020/02/weathervane2-kubernetes.html.

SPECint

Measure compute-intensive performance

Part of the SPEC CPU 2017 benchmark, SPECrate 2017 Integer (also often known by the shorthand name, SPECint) is a suite of tests that stress the processor, memory, and compilers of a solution to provide a general view of compute-intensive performance. It executes a variety of workloads, including artificial intelligence, general data compression, video compression, discrete event simulation, route planning, and more. SPECint generates two higher-is-better throughput scores, one reflecting base performance and the other reflecting peak performance.

SPECfp

Assess floating point performance

Part of the SPEC CPU 2017 benchmark, SPECrate 2017 Floating Point (also often known by the shorthand name, SPECfp) is a suite of tests that stress the processor, memory, and compilers of a solution to provide insight into floating point performance. Its 13 floating point workloads include explosion modeling, fluid dynamics, molecular dynamics, weather forecasting, computational electromagnetics, and regional ocean modeling.¹ SPECfp generates two higher-is-better throughput scores, one reflecting base performance and the other reflecting peak performance.

1. "SPEC CPU®2017 Overview / What's New?", accessed January 17, 2023, https://www.spec.org/cpu2017/Docs/overview.html#suites.

LAMMPS

Measure high performance computing (HPC) capabilities

LAMMPS (an acronym for Large-scale Atomic/Molecular Massively Parallel Simulator) is an open-source molecular dynamics tool for assessing a solution’s performance in this area. The tool includes several benchmark tests, some targeting the CPUs and some targeting a system’s GPU. LAMMPS output varies depending on the benchmark you choose.

Aerospike Database

Assess Aerospike Database performance

Aerospike Database is a NoSQL database platform that supports multiple deployment options on premises or in the cloud. According to Aerospike, the platform "delivers predictable performance, scales from gigabytes to petabytes, is strongly consistent, with unparalleled cross-datacenter replication for a true globally distributed real-time database."¹ To assess overall Aerospike Database performance, we use the Aerospike C benchmark, which Aerospike builds into the Aerospike platform. Aerospike C lets users insert records into the database and run tests with a variety of read/write mixes and thread counts.

1. Aerospike, "Real-time, multi-model NoSQL database," accessed January 18, 2023, https://aerospike.com/products/database/.

Video AI

Evaluate system performance using video AI

For creatives that use AI to improve video quality and resolution, Topaz Video AI may be their software of choice. According to Topaz Labs, their solution “focuses solely on completing a few video enhancement tasks really well: deinterlacing, upscaling, and motion interpolation.”¹ Topaz Video AI includes a built-in benchmark, which outputs scores that reflect a system’s efficiency, processing time, and frames-per-second rates as it processes sets of of videos at different resolutions.

1. "Topaz Video AI," accessed July 5, 2023, https://www.topazlabs.com/topaz-video-ai.

Random Forest

See how well a solution performs using Random Forest to make predictions

According to HiBench, “Random forests (RF) are ensembles of decision trees. Random forests are one of the most successful machine learning models for classification and regression. They combine many decision trees in order to reduce the risk of overfitting.”¹ An organization could use random forests to increase the accuracy of a decision tree; for example, a bank might use RF to make credit risk predictions. Using the HiBench RF workload, we can measure a solution’s throughput while the workload is running and how long it takes a solution to complete the workload.

1. “HiBench,” accessed August 18, https://github.com/Intel-bigdata/HiBench.

WordPress

Quantify website performance

WordPress is a popular open-source content management system based on PHP and MySQL database, which we can use to represent a web server workload. To see how well a given solution performs when running WordPress, we use benchmarks to simulate a large number of users accessing a website. These include the open-source website transaction benchmark suite oss-performance and the Apache JMeter load-testing tool. Regardless of benchmark, we typically look at WordPress performance in terms of web server transactions per second, with a higher number indicating better performance.

TeraSort

Measure Hadoop performance

The TeraSort workload, part of the Intel HiBench suite of benchmarking software, determines how quickly Hadoop clusters can sort a dataset. It combines network, I/O, and compute tasks to provide insight into how a cluster might handle general-purpose jobs on Hadoop. TeraSort generates a lower-is-better score that represents the time to complete a workload and a higher-is-better throughput score.

VMFleet

Assess storage performance

VMFleet is a set of scripts that Microsoft developed to test a solution’s storage performance. With VMFleet, the user deploys multiple VMs, each running a storage load generator, DISKSPD. According to Microsoft, "DISKSPD is a tool that you can customize to create your own synthetic workloads, and test your application before deployment."¹

1. Microsoft, "Use DISKSPD to test workload storage performance," accessed January 18, 2023, https://learn.microsoft.com/en-us/azure-stack/hci/manage/diskspd-overview.

WordCount

Evaluate how well a system performs MapReduce analysis

Part of the Intel HiBench suite of benchmarking software, WordCount tallies the occurrence of each word in a randomized dataset. This CPU-intensive workload makes use of MapReduce, a common Apache Hadoop framework organizations use to access big data. HiBench says that it represents real-world MapReduce tasks, as it extracts “a small amount of interesting data from [a] large data set.”¹With WordCount, we can quantify both the throughput a solution analyzes and how long it takes the solution to complete the workload.

1. "HiBench," accessed August 18, https://github.com/Intel-bigdata/HiBench.

Data reduction testing

Determine data compression and deduplication ratios

Data center sprawl can be very costly, so storage solutions that minimize the amount of space your data occupies are increasingly valuable. In our data reduction testing, we measure before-and-after storage usage to evaluate the compression and deduplication capabilities of different storage solutions.

Decision support workload

Measure how well a solution can handle a decision support system

To test a solution’s capabilities in this area, we often use a workload derived from the TPC-DS benchmark, which simulates a decision support system. Decision support applications analyze large quantities of information and utilize that data to offer possible next steps, making them valuable to organizations across a broad variety of industries.

This workload outputs results in query response time: how long it takes to complete a set of queries of a certain size. (The workload is not a full implementation of official TPC-DS standards, so any results are not directly comparable to published TPC-DS results.)

Video encoding

Quantify intensive video encoding performance

Video files can be extremely large. Compressing a video file so it’s compatible with web and mobile players—while still maintaining picture quality—is called encoding.¹ This important work is also one of the most resource-intensive tasks for a system,² stressing memory, CPU, and GPU. Using a benchmark from HandBrake, a popular converting and encoding platform, we can measure both how long it takes a system to encode a video and the rate of frames per second (FPS) a system handles.

1. “Video Encoding: The Definitive Guide,” accessed October 9, 2023, https://www.dacast.com/blog/what-is-video-encoding/.
2. “HandBrake 1.6.0,” accessed October 9, 2023, https://handbrake.fr/docs/en/1.6.0/technical/performance.html.

Power efficiency testing

Determine performance-per-watt metrics

The server selection process might initially focus on reliability and performance, but energy consumption is an increasingly important consideration. A server that delivers more work for each watt of electricity it consumes can translate to lower data center power and cooling expenses. In our power efficiency testing, we measure a solution’s energy usage and combine it with performance data to determine performance-per-watt metrics that help buyers make informed decisions.

VM and container density testing

Determine how many virtual machines or containers a solution can support

By selecting solutions that achieve greater VM or container density, companies maximize the value of virtualization and can save by being able to perform a given amount of work with a smaller investment. In our density testing, we typically select a workload, such as an OLTP database, and measure the performance of a single VM or container running that workload; we then have a baseline. We next add virtual machines or containers until the average per-VM/per-container performance falls well below the baseline. For example, if performance dropped at eight VMs, we would say the solution supported seven virtual machines at this level of performance.

PCMark

Demonstrate device performance in the workplace

PCMark is a benchmark set that, per UL, “features a comprehensive set of tests that cover the wide variety of tasks performed in the modern workplace.”¹ The main tool (PCMark 10 benchmark) evaluates PCs based on real-world tasks, such as web browsing, video calls, working with spreadsheets, photo editing, and more. Based on a system’s responsiveness while completing the tasks, PCMark delivers scores for each task and category of tasks, as well as an overall score—the higher the score, the smoother the experience end users can expect.

Along with benchmarks for applications and storage, PCMark also features a battery life test that measures how long a worker could expect their device to run on battery

1. “PCMark 10,” accessed February 9, 2023, https://benchmarks.ul.com/pcmark10.

Cinebench

Evaluate PC hardware performance

According to Maxon, “Cinebench offers a real-world benchmark that incorporates a user’s common tasks within Cinema 4D to measure a system’s performance.”¹ Outputting scores for both single-core and multi-core CPU performance, Cinebench measures how a device runs under a high CPU load, allows you to gauge how well the cooling system works during longer tasks, and tests how the device works with intensive 3D tasks.² Higher scores could indicate faster PC response times on graphics-intensive games, product development and design software, and scientific simulations.

1. “Cinebench,” accessed February 9, 2023, https://www.maxon.net/en/cinebench.
2. “Cinebench,” accessed February 9, 2023, https://www.maxon.net/en/cinebench.

MobileMark

Put battery life and device performance to the test

Designed to run on a PC device while it’s unplugged, MobileMark simulates a worker performing productivity and creativity tasks on applications such as Microsoft 365 apps, Google Chrome, and Adobe Creative Cloud. The benchmark reports battery life in hours and minutes while performing these tasks, a battery performance score, and an index score that shows the balance between battery life and performance.¹

1. “An Overview of MobileMark 25,” accessed February 9, 2023, https://bapco.com/wp-content/uploads/2022/11/bapco_mobilemark_25_white_paper_v1.2.pdf.

Migration studies

Assess the ease of migration between solutions

As organizations periodically update and expand their technology infrastructures, the easier it is to migrate from an existing older solution to a new one, the better. We perform migrations the way they happen in the real world, measuring the time and effort they take between different solutions or using different migration or management tools. We can do this for both small-scale laptop changeovers and large-scale, complex upgrades from, for example, one storage array or server cluster to another.

CrossMark

Evaluate the user experience

How quickly a device completes workloads (performance) and responds to a user’s input (responsiveness) both affect user experience. The CrossMark benchmark uses models of real-world applications to measure performance and responsiveness and assigns an overall score it bases on both. According to BAPCo, developers of CrossMark, it is a “cross-platform benchmark that enables direct comparison of desktops and touch enabled devices across Windows, iOS or macOS, and Android.”¹

1. “CrossMark,” accessed February 10, 2023, https://bapco.com/products/crossmark/.

PugetBench

Measure Adobe Creative Cloud capabilities

The Adobe Creative Cloud suite comprises applications that place different kinds of frequently heavy demands on compute resources. It can be challenging for creative professionals and those who buy gear for them to understand how well a given workstation can support these applications. Puget Systems has developed the PugetBench suite of benchmarks “to thoroughly test many of Adobe’s most popular applications using real-world projects and workflows.”¹

The suite includes PugetBench for Photoshop, PugetBench for Lightroom Classic, PugetBench for Premiere Pro, and PugetBench for After Effects.

1. “PugetBench for Adobe Creative Cloud,” accessed February 10, 2023, https://www.pugetsystems.com/labs/articles/PugetBench- for-Adobe-Creative-Cloud-1642/.

WebXPRT

Quantify browser performance

WebXPRT 4 is an industry-standard browser benchmark that compares the performance of web-enabled devices when executing real-world tasks. It contains HTML5, JavaScript, and WebAssembly-based scenarios that mirror activities users perform: Photo Enhancement, Organize Album Using AI, Stock Option Pricing, Encrypt Notes and OCR Scan using WASM, Sales Graphs, and Online Homework.

Principled Technologies is the publisher of the XPRT family of benchmarks and the administrator of the BenchmarkXPRT Development Community.

Procyon benchmark suite

Gauge device performance for professional users

The Procyon benchmark suite from UL comprises five benchmarks that target “professional users in industry, enterprise, government, retail and press.”¹ The Office Productivity Benchmark can measure Windows or macOS device performance for office productivity work using Microsoft Office apps. Also available for Windows or macOS, the Photo Editing Benchmark uses Adobe Lightroom to import, process, and modify a selection of images, then applies multiple edits and layer effects to a photograph in Adobe Photoshop. The AI Inference Benchmark uses neural network models such as MobileNet V3, Inception V4, SSDLite V3, and DeepLab V4. The Battery Life Benchmark measures the battery life of Windows laptops, notebooks, and tablets across different scenarios. And lastly, the Video Editing Benchmark measures the time it takes a PC to apply edits, adjustments, and effects and then export video project files to common formats in Adobe Premiere Pro. These latter three benchmarks are currently Windows-specific, but UL says macOS support is on the way.

1. UL, “UL Procyon benchmark suite,” accessed May 19, 2023, https://benchmarks.ul.com/procyon.

SYSmark

Measure system performance

According to BAPCo, developers of SYSmark, the benchmark measures system performance using “real-world applications and workloads.”¹ It generates an overall rating based on a combination of results that measure system performance while running office and media tasks and scenarios that model common pain points, such as launching files and multitasking. We can run the latest version, SYSmark 30, or the second-to-latest version, SYSmark 25, depending on your preferences.

1. “SYSmark 30,” accessed March 10, 2023, https://bapco.com/products/sysmark-30/.

Geekbench

Measure the single- and multi-core power of a system’s processor

Geekbench is a cross-platform utility that aims to measure the single-core and multi-core power of the central processing unit of a computer. Primate Labs, developer of Geekbench, says the latest version, Geekbench 6, “measures performance in new application areas including Augmented Reality and Machine Learning.”¹

1. Primate Labs, “Introducing Geekbench 6,” accessed March 8, 2023, https://www.geekbench.com/.

Maxon Redshift

Measure GPU-accelerated rendering performance

According to Maxon, Redshift is “a powerful GPU-accelerated renderer, built to meet the specific demands of contemporary high-end production rendering. Tailored to support creative individuals and studios of every size, Redshift offers a suite of powerful features and integrates with industry standard CG applications.”¹ We test a system’s performance by measuring how long the system takes to complete a RedShift render.

1. Maxon, “Redshift,” accessed February 10, 2023, https://www.maxon.net/en/redshift.

Blender

Assess CPU and GPU performance

The Blender benchmark measures CPU and GPU performance by determining the number of samples per minute a system can handle. Blender Open Data is a platform the Blender community provides to collect, display, and query the results of hardware and software performance tests.¹

1. Blender, “About,” accessed February 10, 2023, https://opendata.blender.org/about/.

Onshape

Measure CAD capabilities

Onshape is a computer-aided design software system, available using a software-as-a-service model. The Onshape online performance check provides the browser and GPU with an increasing workload and reports the device limits when the lines and triangles per second framerates start to drop.¹ The higher the framerate a device can support, the better the CAD performance users can expect.

1. Onshape forum, “Performance test results,” accessed February 10, 2023, https://forum.onshape.com/discussion/5225/performance-test-results.

Speedtest

Speedtest by Ookla tests the speed of devices’ connections to the internet. It outputs results in megabits per second (Mbps) for both downloads and uploads. To ensure that any differences in connection speed are due to the devices rather than the internet service provider or other external networking elements, when using this test for multiple devices, we run tests on all devices at the same time and in the same networking environment.

PassMark

Assess multiple elements of PC performance

PassMark offers a number of benchmarks for performance testing of Windows and Linux systems. Among others, these include BurnInTest, which quickly stress-tests PCs; PerformanceTest, which combines 28 speed tests for easy benchmarking of CPU, 2D and 3D graphics, disk, and memory; and MemTest86, which stress-tests a system’s memory.¹ Output depends on which PassMark benchmark you choose.

1. PassMark Software, “Software Products,” accessed February 13, 2023, https://www.passmark.com/products/index.php.

HDXPRT

Test how Windows 10 PCs handle media and more

HDXPRT 4 is a benchmark that assesses the performance of Windows 10 devices by measuring how well they handle real-world media tasks. HDXPRT 4 uses real commercial applications, such as Photoshop and MediaEspresso, to carry out tasks based on photo editing, video conversion, and music editing.

Principled Technologies is the publisher of the XPRT family of benchmarks and the administrator of the BenchmarkXPRT Development Community.

MobileXPRT

Assess Android device performance

MobileXPRT 3 is a benchmark for evaluating the capabilities of Android devices. It runs six performance scenarios: Apply Photo Effects, Create Photo Collages, Create Slideshow, Encrypt Personal Content, Detect Faces to Organize Photos, and Scan Receipts for Spreadsheet. In addition to scoring each scenario, it generates a single overall performance score.

Principled Technologies is the publisher of the XPRT family of benchmarks and the administrator of the BenchmarkXPRT Development Community.

CrXPRT

Measure Chromebook performance and battery life

The CrXPRT 2 benchmark evaluates the performance and battery life of Chromebooks. The CrXPRT performance test measures how quickly a Chromebook handles everyday tasks such as playing video games, watching movies, editing pictures, and doing homework. It generates an overall score and individual scores for each workload. The CrXPRT battery life test produces an estimated battery life, a separate performance score, and a frames-per-second (FPS) rate for a built-in HTML5 gaming component.

Principled Technologies is the publisher of the XPRT family of benchmarks and the administrator of the BenchmarkXPRT Development Community.

TouchXPRT

See how Windows 10 devices stack up

TouchXPRT is a benchmark for evaluating the performance of Windows 10 devices. It runs tests based on five common scenarios: Beautify Photos, Blend Photos, Convert Videos for Sharing, Create Music Podcast, and Create Slideshow from Photos. TouchXPRT produces results for each of the five test scenarios plus an overall score.

Principled Technologies is the publisher of the XPRT family of benchmarks and the administrator of the BenchmarkXPRT Development Community.

AIXPRT

Evaluate your system’s machine learning inference performance

AIXPRT is an AI benchmark tool that makes it easier to evaluate a system's machine learning inference performance by running common image-classification, object-detection, and recommender system workloads. AIXPRT includes support for the Intel OpenVINO, TensorFlow, and NVIDIA TensorRT toolkits to run image-classification and object-detection workloads with the ResNet-50 and SSD-MobileNet v1 networks, as well as a Wide and Deep recommender system workload with the Apache MXNet toolkit. The test reports FP32, FP16, and INT8 levels of precision. Test systems must be running Ubuntu 18.04 LTS or Windows 10, and the minimum CPU and GPU requirements vary by toolkit.

Principled Technologies is the publisher of the XPRT family of benchmarks and the administrator of the BenchmarkXPRT Development Community.

Task-based time testing

See how systems perform real-world tasks

A user’s real-world experience of their system depends a great deal on how long it takes to accomplish the tasks they do every day. We select tasks representative of real-world workflows—such as booting a laptop, opening a document, or exporting a photo—and use hand timers to measure how many seconds a system requires to complete them. We can perform this testing with any task or series of tasks, making it a flexible, realistic approach to highlighting how a system could boost (or harm) productivity.

Thermal testing (end-user)

Discover surface temperatures while idle or under stress

Especially for hybrid or remote workforces who may use devices on their laps, a notebook’s surface temperature can be an important consideration. Excessive heat can be a matter of discomfort or, in extreme cases, could spell hardware failure. By measuring devices’ surface temperatures, we can demonstrate which might offer a more comfortable user experience or be less susceptible to overheating. We can perform this testing while a machine is at rest or under load and supplement the data with thermal camera images.

Thermal testing (enterprise)

See which systems run cooler under load

For organizations with large data centers, power and cooling can be a significant expense. In addition to assessing the quality of power management software tools and a system’s energy efficiency (see “Manageability testing studies” and “Power efficiency testing”), we can measure how well a device’s cooling mechanisms are functioning by assessing how much heat the device is outputting and whether that heat is having an effect on the device’s performance. Depending on the exact tests we run, we may be able to use that data to determine how much money an organization might save on power and cooling by choosing that device.

3DMark

Evaluate graphics and compute performance with demanding gaming workloads

According to UL Solutions, maker of 3DMark, “3DMark is used by millions of gamers, hundreds of hardware review sites, and many of the world's leading manufacturers.”¹ 3DMark offers a suite of benchmarks, each of which runs intensive gaming workloads that put a system’s graphics and compute capabilities to the test. While it outputs useful results for gamers, 3DMark can also help indicate system performance while running compute-intensive workloads that deal with 3D objects.

1. UL Solutions, “3DMark user guide,” accessed March 9, 2023 https://support.benchmarks.ul.com/support/solutions/articles/44002146295-3dmark-user-guide

Acoustic testing

Assess how much noise a device emits

In office environments, background noise can disrupt users’ focus and even their memory.¹ While noise-canceling headphones or closed doors can help, these measures could also leave users feeling isolated. Instead, to cut down on distracting noises, businesses might consider quieter devices for their employees. We test devices’ noise outputs while those devices are idle or running an intensive workload. As another useful point of comparison, we can present these results in conjunction with real-world sound levels. (For more on our audio testing capabilities, see “Microphone quality testing” and “Speaker quality testing”.)

1. Simon Danbury and Dianne C. Berry, “Disruption of office-related tasks by speech and office noise,” accessed March 9, 2023 https://bpspsychub.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2044-8295.1998.tb02699.x

Feature comparison

Compare features on multiple devices

Performance and battery life are critical concerns for buyers of end-user devices, but they’re not the only elements customers consider. We compare devices’ features in a host of areas, including size, weight, screen resolution, number and type of ports, camera and microphone specs, touchscreen and pen-and-ink options, convertibility, and much more. For any feature your device boasts, we can highlight its utility and functionality against the equivalent feature(s) from the competition.

Application verification

Verify apps are working as they should

If a user’s everyday work and/or their hobbies rely on specialty applications—such as CAD apps, engineering applications, or compute-intensive games—they need to know that those applications will run smoothly on any device they might purchase. We can install, open, and complete tasks in any application on any system to verify that the system can handle the application(s) and will function as a user would expect.

Speedometer

See how quickly web-based applications respond to user input

Speedometer is a browser-based responsiveness test for web applications. The benchmark simulates user actions in a demo app, then measures the time it takes to complete those tasks. A higher Speedometer score indicates faster responsiveness, which could translate to a smoother user experience.

Battery life testing

Test how long a device’s battery lasts under varying conditions

Battery life is a top concern for many buyers of end-user devices, whether the device is for business or personal use. We can assess a device’s battery life using a benchmark, such as PCMark or MobileMark, or by continuously running a real-world workload, such as local or streaming video playback, and measuring how long the device’s battery lasts before dying. We can also use this data to assess the efficiency of the battery in minutes per WHr.

Bayesian classification (HiBench)

Measure ML data classification performance

The Bayesian classification workload from the HiBench suite is a machine learning (ML) workload that measures a solution’s ability to mine data to make probabilistic predictions. The training portion of Naïve Bayesian classification is a popular algorithm for knowledge discovery and data mining, with Bayesian classifiers performing important tasks such as identifying and filtering spam emails. In using this workload for performance testing, we measure throughput while the workload is running and how long it takes a solution to complete the workload.

Unigine benchmarks

Measure compute-intensive performance for gaming

The Unigine performance benchmarks “generate true in-game rendering workloads across multiple platforms” and are useful for assessing “the stability of PC hardware (CPU, GPU, power supply, cooling system) under extremely stressful conditions, as well as for overclocking.”¹ The three benchmarks—Superposition, Valley, and Heaven—incorporate interactivity and large maps of detailed 3D terrain, making them suitable for assessing a device’s potential gaming and/or 3D rendering performance.

1. Unigine, accessed March 15, 2023 https://benchmark.unigine.com/.

In-game benchmarks

Assess devices’ real-world gaming performance

If one of a PC user’s primary concerns is gaming performance, what better way to assess the device than using a benchmark built into a game? Several PC games, including Sid Meier’s Civilization IV and Deus X: Mankind Divided, include built-in benchmarks that measure metrics such as AI turn speed and frame rate. These benchmarks can also be valuable for an audience that does not prioritize gaming, allowing us to assess how a device performs under a very graphics-intensive workload.

Basemark GPU

Evaluate a device’s graphics performance

Basemark GPU, which Basemark calls “the ultimate graphics performance benchmark,” evaluates graphics performance on Windows, macOS, Ubuntu, Linux, Android, and iOS devices.¹ It does this by running through a graphically intensive scene and generating a score that users can compare to other Basemark scores. It offers two modes: High Quality, for desktop systems, and Medium Quality, for mobile systems.

1. Basemark, “Basemark GPU,” accessed March 6, 2023 https://www.basemark.com/benchmarks/basemark-gpu/.

Durability testing (end-user)

Discover how a device holds up under wear and tear

Many users primarily rely on their devices at home and in the office, packing them carefully into backpacks for the transition. But others need a device that can stand up to wear and tear—environments as varied as hospitals, construction sites, and classrooms all require tech that delivers both high performance and strong durability. We can test a laptop’s or tablet’s toughness from multiple perspectives, including how it handles drops, spills, scratches, heavy rain, bright sunlight, extreme temperatures, and more.

Thermal testing (enterprise)

See which systems run cooler under load

For organizations with large data centers, power and cooling can be a significant expense. In addition to assessing the quality of power management software tools and a system’s energy efficiency (see “Manageability studies” and “Power efficiency testing”), we can measure how well a device’s cooling mechanisms are functioning by assessing how much heat the device is outputting and whether that heat is having an effect on the device’s performance. Depending on the exact tests we run, we may be able to use that data to determine how much money an organization might save on power and cooling by choosing that device.

Deployment studies (on end-user devices)

Quantify and qualify the benefits of automated end-user device deployment

Deploying desktops and laptops to hundreds or thousands of employees has never been a straightforward task for IT, and today’s increasingly remote or hybrid environments have only added complexity. We can test your deployment tools and services the way real IT teams would use them, assessing how much time and effort they save compared to a manual deployment process or competing solution. Our reports incorporate in-depth discussions of our experience using your solution, focusing on benefits to both IT and end-users.

K-means

Assess machine learning capabilities to cluster data

Using workloads from the HiBench or Spark-Bench benchmark suites, we measure how well a system completes k-means clustering. As a machine learning algorithm, k-means clustering sorts data into similar groups, helping businesses analyze data to map new storefront locations or target marketing to different demographics. The sooner a solution completes this compute-intensive work, the sooner you can use the resulting data for business-critical endeavors. With this test, in which we have the option to customize size of the dataset, we report how long it takes a solution to complete the workload and what throughput the solution delivers while doing so.

SPECviewperf

Quantify graphics performance with workloads based on professional applications

For systems running under OpenGL and DirectX application programming interfaces, SPECviewperf is “the worldwide standard for measuring graphics performance based on professional applications,” according to the SPEC organization.¹ Running rendering workloads based on applications such as 3ds Max, Maya, and Solidworks, SPECviewperf outputs a score where higher numbers indicate better performance.

1. Standard Performance Evaluation Corporation, “SPECviewperf® 2020 v3,” accessed March 9, 2023 https://gwpg.spec.org/benchmarks/benchmark/specviewperf-2020-v3-1/.

SPECworkstation

Assess workstation performance with a variety of workloads

The SPECworkstation benchmark suite tests a device’s processor, graphics card, I/O, and memory bandwidth performance across a series of 30 workloads containing almost 140 total tests. In addition to general tasks, workload categories include tests specific to industries such as product development, financial services, energy (oil and gas), and more. The SPEC organization modeled tests on real-world applications relevant to key sectors, such as the Blender and Maya apps for the media and entertainment industries.¹ The higher each workload score, the faster you might expect a device to perform for employees doing that type of work.

1. Standard Performance Evaluation Corporation, “SPECworkstation® 3.1,” accessed March 9, 2023 https://gwpg.spec.org/benchmarks/benchmark/specworkstation-3_1/.

Backup and restore studies (end-user)

Measure the efficacy of any end-user backup and recovery solution

Losing laptop data can be a logistical and personal nightmare—not to mention a major impact to productivity at work. We can test the efficacy, ease of use, and features of your backup and recovery services and your competitors’, proving time savings and exploring your solution’s benefits to users and organizations alike.

Microphone quality testing

In an environment where many users work remotely or in open office environments, the quality of a system’s onboard microphone may become especially important. We can assess microphones by testing their off-axis rejection and/or noise reduction capabilities, both measurements that quantify how much ambient noise makes it through the microphone while a speaker is talking. For users who work in busy offices or loud environments, these metrics may be particularly useful: When they need to hop on a video call, the person on the other end should be able to hear their voice with clear, focused sound and minimal distracting background noises. A mic with better off-axis rejection may also reduce the need for a separate headset device.

Speaker loudness & quality testing

While some folks use headphones or external speakers, many people prefer to utilize their device’s built-in speakers. In a world where many of us are video-conferencing every day, speaker quality is important. We can measure how loud a speaker can get, with louder speakers being better for the user. (You can always turn a speaker down, but you can’t turn it up past its maximum point!) We can also pair this test with a jury study on speaker quality, asking a set of sample users what they like and dislike about the sound of the speakers. Together, the objective and subjective approaches paint a holistic picture of the experience of using a system’s speakers.