TARE: Tail Aware Evaluation of HPC Job Runtime Prediction
TL;DR
An empirical evaluation methodology for HPC job runtime prediction that focuses on the tail, combining GeoAccuracy weighted by resource usage with decile and split analyses and UserReq achieves the highest GeoAccuracy and lowest underestimation rate on all three datasets.
Abstract
Runtime estimates affect reservation quality, backfilling opportunities, and queue delay in HPC schedulers. Under heavy tailed workloads, however, averaging over jobs can misrepresent scheduling impact because a small fraction of jobs dominates resource usage. This paper presents an empirical evaluation methodology for HPC job runtime prediction that focuses on the tail, combining GeoAccuracy weighted by resource usage with decile and split analyses. Using production traces from NREL Eagle and ALCF Mira/Intrepid, we compare XGBoost and Last2 against the user provided walltime estimate at submission (UserReq). Across all three datasets, evaluation focused on the tail changes the offline conclusion: MeanAccuracy keeps the methods relatively close, whereas GeoAccuracy reveals clearer separation and makes UserReq's strength in the upper tail visible. In the top decile, UserReq achieves the highest GeoAccuracy and lowest underestimation rate on all three datasets, and this pattern remains stable across rolling splits. We then translate this signal into a simple hybrid scheduling policy that keeps XGBoost for most jobs and routes the top decile by proxy_cost at submission to UserReq. Online replay on four production queues reduces mean wait time by up to 8% and increases backfilled jobs by 50% to 115%. These results show that offline evaluation focused on the tail better characterizes prediction quality relevant to scheduling and informs scheduling policy design.