Beyond ROC-AUC: Decision-Adjusted Analytical Value for Capacity-Constrained Management Analytics under Temporal Distribution Shift

Abstract

Management analytics is commonly evaluated with model-centric statistics such as the area under the receiver operating characteristic curve (ROC-AUC), yet managerial deployment converts scores into costly actions under finite capacity, changing base rates, and asymmetric economic consequences. This study introduces Decision-Adjusted Analytical Value (DAAV), a normalized framework that evaluates a predictive policy against random-capacity and attainable-oracle benchmarks after incorporating an economic action threshold and an operational capacity constraint. The framework is tested on the UCI Bank Marketing dataset using 41,188 chronologically ordered campaign records. To prevent post-contact leakage, call duration is excluded. Logistic regression, random forest, and Extreme Gradient Boosting (XGBoost) models are trained on the first 75% of observations, calibrated on the next 10%, and evaluated on the final 15% temporal holdout. Performance is assessed across 20 benefit-cost and capacity scenarios, with paired nonparametric bootstrapping for the focal policy comparison. The temporal holdout exhibits a pronounced prevalence shift, from 6.8% in validation to 38.5% in testing. Random forest attains the highest ROC-AUC (0.653), but is not the value-maximizing model in 16 of 20 operating scenarios. At a benefit-cost ratio of 3 and 50% capacity, logistic regression produces 226.9 normalized net-value units per 1,000 cases versus 95.6 for random forest; the paired difference is 131.0 units (95% bootstrap interval: 110.4 to 151.3). Static Platt calibration fitted before the shift further reduces downstream value despite leaving discrimination unchanged. The findings show that ranking models by predictive accuracy can misallocate managerial attention when economic thresholds and resource constraints bind. DAAV provides managers, analytics teams, and governance functions with a transparent, decision-centered complement to conventional predictive metrics for model selection and deployment under temporal shift.