Baseliner: A Plugin-Oriented, Hardware-Agnostic GPU Benchmarking Library
TL;DR
Empirical evaluation on NVIDIA V100 and AMD MI210 hardware shows Baseliner reaches measurement parity with vendor libraries within the precision floor of GPU event timing while remaining competitive in wall-clock time.
Abstract
GPU benchmarking underpins performance claims across high-performance computing (HPC), yet the software infrastructure producing these measurements is rarely treated as a first-class research artifact. Existing tools fragment into benchmark suites where measurement logic is coupled to kernels, vendor profilers designed for diagnostic tracing rather than statistical benchmarking, and ecosystem-specific libraries that lack cross-hardware portability. We present Baseliner, a hardware-agnostic GPU benchmarking framework whose plugin-oriented architecture decouples workload definition from measurement orchestration. A formalized Workload Interface and a backend abstraction layer let workloads, timing backends, stopping criteria, and statistics engines be swapped independently of the core engine. An adapter mechanism further enables a single Workload definition to be executed by Baseliner, NVBench, or PrimBench, eliminating redundant implementations across toolchains. We validate the framework through blas-baseliner, a cross-vendor BLAS suite spanning cuBLAS and rocBLAS. Empirical evaluation on NVIDIA V100 and AMD MI210 hardware shows Baseliner reaches measurement parity with vendor libraries within the precision floor of GPU event timing while remaining competitive in wall-clock time. Video: https://youtu.be/LdKUEqewmQw