The Fourth Quadrant: A Stylized View of Benign Misfitting

Abstract

Training error is what we can observe on a training set; test error is the quantity we care about. We study squared-error linear regression in a deterministic $(d+1)$-dimensional single-spike model. Each training vector has spike coordinate $\sqrt{\gamma}$, $\gamma>1$, and equal-norm, mutually orthogonal nuisance components. Training labels are $1$. Fresh test points are centered Gaussian with covariance $\operatorname{diag}(\gamma,1,\ldots,1)$ and noise-free label $x[1]/\sqrt{\gamma}$. We compute the exact train--test tradeoff for linear predictors in the training span. When the spike direction is unknown, allowing all linear predictors does not improve this tradeoff, on average or in the worst case over directions consistent with the data. There is a range of training-set sizes where every span predictor that generalizes well must fit the training data worse than the zero predictor. We call this benign misfitting, or the fourth quadrant. At its sharpest, test error tends to zero while training error diverges, with both zero-predictor errors normalized to one. The best span predictor begins to generalize when $n\gg d/\gamma^2$, while interpolation needs $n\gg d/\gamma$. In the intervening window $d/\gamma^2\ll n\ll d/\gamma$, useful span prediction lies beyond interpolation: training predictions overshoot the labels. One-pass stochastic gradient descent with a large constant learning rate attains small test error throughout this window, matching the best span predictor up to a logarithmic factor, while leaving large empirical training error. Finally, the nuisance component responsible for the training misfit also controls adversarial sensitivity.