Learning to Doubt: Forgetting Aware Learning for Neural Networks

Abstract

Modern neural networks are often miscalibrated, assigning high confidence to predictions they have not learned stably, which leads to overconfident errors under noise, imbalance, and distribution shift. We propose Forgetting-Aware Learning (FAL) using a simple ranking regularizer that leverages forgetting events during training as a temporal signal of epistemic uncertainty. The proposed regularizer penalizes high confidence on volatile samples, enforcing an isotonic relationship between confidence and learning stability and thereby reducing the mass of high-confidence mistakes that dominate calibration error. We show that FAL provides complementary benefits to Confidence-Aware Learning (CAL) using temporal instability as a regularization signal yielding a more self-aware model. Empirically, FAL improves calibration and confidence ranking while preserving accuracy. We also show that combining FAL and CAL with focal loss yields the strongest calibration in our study, significantly outperforming cross-entropy-based counterparts in the expected calibration error (ECE). We validate these findings across five popular image benchmark datasets, (with additional results on tabular data reported in the appendix), demonstrating that confidence should be earned by stable learning.