Multi-program benchmark definition

TL;DR: This work proposes a four-tuple that formally defines multi-program benchmarks in a well-defined way, and examines how four different classes of benchmarks created by varying the elements of this tuple align with real-world use-cases.

Abstract: Although definition of single-program benchmarks is relatively straight-forward—a benchmark is a program plus a specific input—definition of multi-program benchmarks is more complex. Each program may have a different runtime and they may have different interactions depending on how they align with each other. While prior work has focused on sampling multiprogram benchmarks, little attention has been paid to defining the benchmarks in their entirety. In this work, we propose a four-tuple that formally defines multi-program benchmarks in a well-defined way. We then examine how four different classes of benchmarks created by varying the elements of this tuple align with real-world use-cases. We evaluate the impact of these variations on real hardware, and see drastic variations in results between different benchmarks constructed from the same programs. Notable differences include significant speedups versus slowdowns (e.g., +57% vs -5% or +26% vs -18%), and large differences in magnitude even when the results are in the same direction (e.g., 67% versus 11%).