David Kanter

TL;DR: The current landscape of TinyML is presented and the challenges and direction towards developing a fair and useful hardware benchmark for TinyML workloads are discussed, along with three preliminary benchmarks and the selection methodology are discussed.

TL;DR: MLPerf as discussed by the authors is an ML benchmark that overcomes three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time-to-solution exhibits high variance.

TL;DR: The design choices behind MLPerf, a machine learning performance benchmark that has become an industry standard, are described, showing growing adoption and improvements to software-stack performance and scalability.

TL;DR: DataPerf is presented, a benchmark package for evaluating ML datasets and dataset-working algorithms to enable the “data ratchet,” in which training sets will aid in evaluating test sets on the same problems, and vice versa, to generate a virtuous loop that will accelerate development of data-centric AI.