BenchmarkXPRT Blog banner

Category: image classification

A clearer picture of WebXPRT 4

Justin Greene
By Justin Greene
on October 7, 2021

The WebXPRT 4 development process is far enough along that we’d like to share more about changes we are likely to make and a rough target date for publishing a preview build. While some of the details below will probably change, this post should give readers a good sense of what to expect.

General changes

Some of the non-workload changes in WebXPRT 4 relate to our typical benchmark update process, and a few result directly from feedback we received from the WebXPRT tech press survey.

  • We will update the aesthetics of the WebXPRT UI to make WebXPRT 4 visually distinct from older versions. We do not anticipate significantly changing the flow of the UI.
  • We will update content in some of the workloads to reflect changes in everyday technology. For instance, we will upgrade most of the photos in the photo processing workloads to higher resolutions.
  • In response to a request from tech press survey respondents, we are considering adding a looping function to the automation scripts.
  • We are investigating the possibility of shortening the benchmark by reducing the default number of iterations from seven to five. We will only make this change if we can ensure that five iterations produce consistently low score variance.

Changes to existing workloads

  • Photo Enhancement. This workload applies three effects to two photos each (six photos total). It tests HTML5 Canvas, Canvas 2D, and JavaScript performance. The only change we are considering is adding higher-resolution photos.
  • Organize Album Using AI. This workload currently uses the ConvNetJS neural network library to complete two tasks: (1) organizing five images and (2) classifying the five images in an album. We are planning to replace ConvNetJS with WebAssembly (WASM) for both tasks and are considering upgrading the images to higher resolutions.
  • Stock Option Pricing. This workload calculates and displays graphic views of a stock portfolio using Canvas, SVG, and dygraph.js. The only change we are considering is combining it with the Sales Graphs workload (below).
  • Sales Graphs. This workload provides a web-based application displaying multiple views of sales data. Sales Graphs exercises HTML5 Canvas and SVG performance. The only change we are considering is combining it with the Stock Option Pricing workload (above).
  • Encrypt Notes and OCR Scan. This workload uses ASM.js to sync notes, extract text from a scanned receipt using optical character recognition (OCR), and add the scanned text to a spending report. We are planning to replace ASM.js with WASM for the Notes task and with WASM-based Tesseract for the OCR task.
  • Online Homework. This workload uses regex, arrays, strings, and Web Workers to review DNA and spell-check an essay. We are not planning to change this workload.

Possible new workloads

  • Natural Language Processing (NLP). We are considering the addition of an NLP workload using ONNX Runtime and/or TensorFlowJS. The workload would use Bidirectional Encoder Representations from Transformers (BERT) to answer questions about a given text. Similar use cases are becoming more prevalent in conversational bot systems, domain-specific document search tools, and various other educational applications.
  • Message Scrolling. We are considering developing a new workload that would use an Angular or React.js to scroll through hundreds of messages. We’ll share more about this possible workload as we firm up the details.

The release timeline

We hope to publish a WebXPRT 4 preview build in the second half of November, with a general release before the end of the year. If it looks as though that timeline will change significantly, we’ll provide an update here in the blog as soon as possible.

We’re very grateful for all the input we received during the WebXPRT 4 planning process. If you have any questions about the details we’ve shared above, please feel free to ask!

Justin

Posted in benchmark, Benchmark metrics, Benchmarking, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, Future of performance evaluation, HTML5, image classification, image processing, Performance benchmarking, Web-based testing, WebAssembly, WebXPRT, webxprt 4 | Tagged , , , , |

The AIXPRT learning tool is now live (and a CloudXPRT version is on the way)!

Justin Greene
By Justin Greene
on December 10, 2020

We’re happy to announce that the AIXPRT learning tool is now live! We designed the tool to serve as an information hub for common AIXPRT topics and questions, and to help tech journalists, OEM lab engineers, and everyone who is interested in AIXPRT find the answers they need in as little time as possible.

The tool features four primary areas of content:

  • The Q&A section provides quick answers to the questions we receive most from testers and the tech press.
  • The AIXPRT: the basics section describes specific topics such as the benchmark’s toolkits, networks, workloads, and hardware and software requirements.
  • The testing and results section covers the testing process, metrics, and how to publish results.
  • The AI/ML primer provides brief, easy-to-understand definitions of key AI and ML terms and concepts for those who want to learn more about the subject.

The first screenshot below shows the home screen. To show how some of the popup information sections appear, the second screenshot shows the Inference tasks (workloads) entry in the AI/ML Primer section. 

We’re excited about the new AIXPRT learning tool, and we’re also happy to report that we’re working on a version of the tool for CloudXPRT. We hope to make the CloudXPRT tool available early next year, and we’ll post more information in the blog as we get closer to taking it live.

If you have any questions about the tool, please let us know!

Justin

Posted in AI, AIXPRT, benchmark, BenchmarkXPRT, Cloud, CloudXPRT, computer vision, Cross-platform benchmarks, data analytics, image classification, Machine learning, object detection, OpenVINO, Performance benchmarking, TensorFlow, TensorRT | Tagged , , , , , , |

Potential web technology additions for WebXPRT 4

Justin Greene
By Justin Greene
on August 20, 2020

A few months ago, we invited readers to send in their thoughts and ideas about web technologies and workload scenarios that may be a good fit for the next WebXPRT. We’d like to share a few of those ideas today, and we invite you to continue to send your feedback. We’re approaching the time when we need to begin firming up plans for a WebXPRT 4 development cycle in 2021, but there’s still plenty of time for you to help shape the future of the benchmark.

One of the most promising ideas for WebXPRT 4 is the potential addition of one or more WebAssembly (WASM) workloads. WASM is a low-level, binary instruction format that works across all modern browsers. It offers web developers a great deal of flexibility and provides the speed and efficiency necessary for running complex client applications in the browser. WASM enables a variety of workload scenario options, including gaming, video editing, VR, virtual machines, image recognition, and interactive educational content.

In addition, the Chrome team is dropping Portable Native Client (PNaCL) support in favor of WASM, which is why we had to remove a PNaCL workload when updating CrXPRT 2015 to CrXPRT 2. We generally model CrXPRT workloads on existing WebXPRT workloads, so familiarizing ourselves with WASM could ultimately benefit more than one XPRT benchmark.

We are also considering adding a web-based machine learning workload with TensorFlow for JavaScript (TensorFlow.js). TensorFlow.js offers pre-trained models for a wide variety of tasks including image classification, object detection, sentence encoding, natural language processing, and more. We could also use this technology to enhance one of WebXPRT’s existing AI-themed workloads, such as Organize Album using AI or Encrypt Notes and OCR Scan.

Other ideas include using a WebGL-based workload to target GPUs and investigating ways to incorporate a battery life test. What do you think? Let us know!

Justin

Posted in AI, battery life, benchmark, BenchmarkXPRT, BenchmarkXPRT development community, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, image classification, Machine learning, object detection, Performance benchmarking, Virtual reality, Web-based testing, WebAssembly, WebGL, WebXPRT, WebXPRT 3, What makes a good benchmark? | Tagged , , , , , , , , , , , , |

Make confident choices about your company’s future tech with the XPRTs

Justin Greene
By Justin Greene
on April 23, 2020

Durham, NC, April 23, 2020 — Principled Technologies and the BenchmarkXPRT Development Community have released a video on the benefits of consulting the XPRTs before committing to new technology purchases.

AIXPRT, one of the battery of XPRT benchmark tools, runs image-classification and object-detection workloads to determine how well tech handles AI and machine learning.

CloudXPRT, another XPRT tool, accurately measures the end-to-end performance of modern, cloud-first applications deployed on infrastructure as a service (IaaS) platforms – allowing corporate decision-makers to select the best configuration for every objective.

All of the XPRTs give companies the real-world information necessary to determine which prospective future tech p – and which will disappoint

According to the video, “The XPRTs don’t just look at specs and features; they gauge a technology solution’s real-world performance and capabilities. So you know whether switching environments is worth the investment. How well solutions support machine learning and other AI capabilities. If next-gen releases beat their rivals or fall behind the curve.”

Watch the video at facts.pt/pyt88k5. To learn more about how AIXPRT, CloudXPRT, WebXPRT, MobileXPRT, TouchXPRT, CrXPRT, and HDXPRT can help IT decision-makers can make confident choices about future purchases, go to www.BenchmarkXPRT.com.

About Principled Technologies, Inc.
Principled Technologies, Inc. is the leading provider of technology marketing and learning & development services. It administers the BenchmarkXPRT Development Community.

Principled Technologies, Inc. is located in Durham, North Carolina, USA. For more information, please visit www.principledtechnologies.com.

Company Contact
Justin Greene
BenchmarkXPRT Development Community
Principled Technologies, Inc.
1007 Slater Road, Suite #300
Durham, NC 27703
BenchmarkXPRTsupport@PrincipledTechnologies.com

Posted in AI, AIXPRT, benchmark, Benchmarking, BenchmarkXPRT, Cloud, CloudXPRT, Cross-platform benchmarks, IaaS, image classification, object detection, Performance benchmarking, Press Releases | Tagged , , , , , , |

Thinking ahead to WebXPRT 4

Justin Greene
By Justin Greene
on March 5, 2020

It’s been about two years since we released WebXPRT 3, and we’re starting to think about the WebXPRT 4 development cycle. With over 529,000 runs to date, WebXPRT continues to be our most popular benchmark because it’s quick and easy to run, it runs on almost anything with a web browser, and it evaluates performance using the types of web technologies that many people use every day.

For each new version of WebXPRT, we start the development process by looking at browser trends and analyzing the feasibility of incorporating new web technologies into our workload scenarios. For example, in WebXPRT 3, we updated the Organize Album workload to include an image-classification task that uses deep learning. We also added an optical character recognition task to the Encrypt Notes and OCR scan workload, and introduced a new Online Homework workload that combined part of the DNA Sequence Analysis scenario with a writing sample/spell check scenario.

Here are the current WebXPRT 3 workloads:

  • Photo Enhancement: Applies three effects, each using Canvas, to two photos.
  • Organize Album Using AI: Detects faces and classifies images using the ConvNetJS neural network library.
  • Stock Option Pricing: Calculates and displays graphic views of a stock portfolio using Canvas, SVG, and dygraphs.js.
  • Encrypt Notes and OCR Scan: Encrypts notes in local storage and scans a receipt using optical character recognition.
  • Sales Graphs: Calculates and displays multiple views of sales data using InfoVis and d3.js.
  • Online Homework: Performs science and English assignment tasks using Web Workers and Typo.js spell check.

What new technologies or workload scenarios should we add? Are there any existing features we should remove? Would you be interested in an associated battery life test? We want to hear your thoughts and ideas about WebXPRT, so please tell us what you think!

Justin

Posted in AI, benchmark, Benchmark metrics, Benchmarking, BenchmarkXPRT, Browser-based benchmarks, Collaborative benchmark development, Cross-platform benchmarks, image classification, JavaScript, Performance benchmarking, WebXPRT, WebXPRT 3, What makes a good benchmark? | Tagged , , , |

AIXPRT’s unique development path

Justin Greene
By Justin Greene
on December 12, 2019

With four separate machine learning toolkits on their own development schedules, three workloads, and a wide range of possible configurations and use cases, AIXPRT has more moving parts than any of the XPRT benchmark tools to date. Because there are so many different components, and because we want AIXPRT to provide consistently relevant evaluation data in the rapidly evolving AI and machine learning spaces, we anticipate a cadence of AIXPRT updates in the future that will be more frequent than the schedules we’ve used for other XPRTs in the past. With that expectation in mind, we want to let AIXPRT testers know that when we release an AIXPRT update, they can expect minimized disruption, consideration for their testing needs, and clear communication.

Minimized disruption

Each AIXPRT toolkit (Intel OpenVINO, TensorFlow, NVIDIA TensorRT, and Apache MXNet) is on its own development schedule, and we won’t always have a lot of advance notice when new versions are on the way. Hypothetically, a new version of OpenVINO could release one month, and a new version of TensorRT just two months later. Thankfully, the modular nature of AIXPRT’s installation packages ensures that we won’t need to revise the entire AIXPRT suite every time a toolkit update goes live. Instead, we’ll update each package individually when necessary. This means that if you only test with a single AIXPRT package, updates to the other packages won’t affect your testing. For us to maintain AIXPRT’s relevance, there’s unfortunately no way to avoid all disruption, but we’ll work to keep it to a minimum.

Consideration for testers

As we move forward, when software compatibility issues force us to update an AIXPRT package, we may discover that the update has a significant effect on results. If we find that results from the new package are no longer comparable to those from previous tests, we’ll share the differences that we’re seeing in our lab. As always, we will use documentation and versioning to make sure that testers know what to expect and  that there’s no confusion about which package to use.

Clear communication

When we update any package, we’ll make sure to communicate any updates in the new build as clearly as possible. We’ll document all changes thoroughly in the package readmes, and we’ll talk through significant updates here in the blog. We’re also available to answer questions about AIXPRT and any other XPRT-related topic, so feel free to ask!

Justin

Posted in AI, AIXPRT, benchmark, BenchmarkXPRT, Collaborative benchmark development, Cross-platform benchmarks, Future of performance evaluation, image classification, Machine learning, OpenVINO, Performance benchmarking, TensorRT | Tagged , , , , , , , |

Check out the other XPRTs:

Forgot your password?
© Principled Technologies, Inc. 2025   |   Privacy policy

Cookie Notice: Our website uses cookies to deliver a smooth experience by storing logins and saving user information. By continuing to use our site, you agree with our usage of cookies as our privacy policy outlines.

x