Beyond Single-Level PGO: Understanding and Harnessing Cross-Level Optimization Effects in Industrial Systems
TL;DR
A large-scale industrial study of multi-level PGO is presented and a practical guideline for applying PGO safely across system layers is proposed, which highlights the importance of workload representativeness and cross-layer awareness for achieving reliable performance improvements.
Abstract
Profile-guided optimization (PGO) is used to improve software performance in applications and operating system kernels. However, prior work applies PGO to a single layer in isolation, overlooking cross-layer interactions in systems, where kernel and application are optimized independently using different workloads. This practice is common in production environments and makes its impact on end-to-end performance important to understand. It introduces a mismatch between training and deployment, where profiling workloads may not reflect the final co-executed system. Despite its prevalence, the implications of such cross-layer mismatch remain poorly understood. In this paper, we present a large-scale industrial study of multi-level PGO across three representative products. We evaluate the combined effects of independently optimized kernel and application under realistic deployment conditions. Our results show that multi-level PGO can yield performance gains of up to 17.04%, but may lead to severe regressions of up to 81.90%. Through performance counter analysis, we attribute these effects to microarchitectural factors such as cache contention and branch misprediction arising from mismatched profiling assumptions. These findings show that PGO benefits are not necessarily compositional across software layers. Based on these insights, we propose Multi-Level PGO (ML-PGO), a practical guideline for applying PGO safely across system layers. Our work highlights the importance of workload representativeness and cross-layer awareness for achieving reliable performance improvements.