Detectable, Task-Relevant, or Harmful? Construct-Valid Evaluation of Unlabeled Distribution-Shift Monitoring in Chest Radiography

Abstract

Distribution-shift alarms do not establish classifier harm. We evaluated the detectability, frozen-head relevance, and performance deterioration of two public DenseNet-121 chest-radiography classifiers under 30 controlled image conditions. Signals covered pixels, 64-component PCA representations, logits, probabilities, and calibrated high-error selection. An exact whole-model-null intervention changed representations while preserving all 18 released output slots to within 4.09 × 10−14; PCA monitoring detected all six interventions, five-logit monitoring detected none, and the paired Brier change was negligible. Across image shifts, PCA and logit alarm rates were 97% and 52%. In held-source/held-family prediction of primary Brier harm, probability mean displacement had the lowest error (MAE 0.00679); neither PCA signal outperformed the training-mean baseline after correction, whereas three output-facing signals beat both prespecified baselines. Secondary results were outcome-dependent: PCA RFF-MMD ranked best for macro-AUC and locked-F1 loss. An exploratory ConvNeXtV2-Atto/MobileViT-XS arm reproduced the construct separation: representation monitoring detected 57/60 conditions and all six task-null controls, while logit monitoring detected no task-null control; probability mean had the lowest held-architecture/held-family Brier-harm error ratio (0.404). The results show that detectability, task relevance, and metric-specific harm are distinct constructs and should be reported separately.