A Framework for Value-Aligned Modeling and Robust Experimentation

Abstract

When developing large-scale industrial recommender systems, a critical challenge is misalignment between offline model evaluation metrics and the final business objective. This disconnect often leads to inefficient development cycles where promising offline improvements fail to translate into online performance gains. This paper introduces a systematic framework to bridge this gap by applying novel value-aligned model evaluation metrics and reducing systematic errors in experiment setups at each decision-making stage. We focus on three experiment stages of the development process: offline experiments, low-traffic online A/B experiments, and high-traffic online A/B experiments. Our results from incorporating advertiser-defined value into a weighted offline objective improves Delivered Value by 4.7% on an O($B) surface and strengthens correlation between offline and low-traffic online results from a baseline of 0.1 (negligible) to now 0.65 (substantial) correlation; and our methods for reducing systematic biases in low-traffic A/B experimentation setups further improve low-traffic to high-traffic metric correlation from 0.53 to 0.93. Validated with data from over 100 large-scale offline and online experiments in both conversion (pConvs) and click (pCTR) prediction models, our framework provides a comprehensive guide for aligning machine learning development with core business objectives, leading to more accurate decision-making and significant (2x) improvements in engineering productivity.