Early Stopping Without Validation Data in Weakly Supervised Learning.

Abstract

Early stopping is a fundamental technique for mitigating overfitting in machine learning and is especially critical in many weakly supervised learning scenarios. Conventional early stopping, however, needs a sufficiently sized and representative validation set. This creates a trade-off: dedicating more data for validation can diminish the training set and limit model performance, while an insufficient validation set may lead to sub-optimal model selection. This challenge is particularly serious in many weakly supervised learning scenarios where validation data may be entirely unavailable or unreliable. In this paper, we propose Label Wave, which does not require validation data for selecting the desired model across various weakly supervised learning paradigms, including learning with noisy labels (LNL), positive-unlabeled learning, and unlabeled-unlabeled learning. It works by tracking the changes in the model's predictions on the training set during the training process, aiming to halt training before the model excessively fits misleading patterns. This method is empirically supported by our observation that minimum fluctuations in predictions typically occur at the training epoch before the model transitions from fitting reliable patterns to misleading ones. Extensive experiments further provide large-scale evaluation for LNL across diverse noise conditions, model families, optimizers, and data modalities. Label Wave selects near-oracle checkpoints (test-accuracy oracle gaps: 0.15-0.93 percentage points in controlled sweeps and 1.90 percentage points on 71 tabular tasks) and boosts 7 LNL methods by 1.98/2.86 percentage points over 10%/20% hold-out validation.