Foundations of Multivariate Learning

Abstract

Many modern scientific studies record more variables than observations. A single experiment may measure thousands of quantities while collecting only hundreds of samples. Inthis regime the classical multivariate toolbox stops working. Principal component analysis,the standard tool for keeping the few strongest patterns of a table of data, loses its guarantees, and so do Gaussian graphical models, which map the direct links between variables,mixture models, which describe data that come from a few hidden groups, and robust estimators, which are built to resist outliers. Deep learning, meanwhile, does very well here. This survey explains why, structure by structure. It first arranges the mathematical foundations into seven layers, from matrix algebra to inference and validation, and traces each deep architecture to the layers that power it. It then reviews the classical theory and its five characteristic failure modes, and shows when depth escapes the curse of dimensionality and how much data is saved when a known symmetry is built into a network. The core of the survey is a taxonomy built on structural assumptions: each deep architecture autoencoders, graph neural networks, attention and state-space models, normalizing flows, diffusion, flow matching, deep clustering, robust autoencoders – is paired with the classical method it extends. Foundation models, which are pretrained once and then applied to a new dataset in a single forward pass, are shown to generalize empirical Bayes shrinkage. A synthesis records, guarantee by guarantee, what classical theory survives in the deep extensions, what weakens, and what is lost. Benchmark evidence, applications, and six open problems complete the survey.