Fractal basin geometry and the limits to predictability and reproducibility in deep learning

Abstract

Fundamental limits to predictability are central to our understanding of many physical and computational systems. In deep learning, training outcomes are notoriously difficult to predict and reproduce, yet the mechanism underlying these difficulties has remained unclear. Here we propose fractal, riddled basin geometry as a unifying dynamical-systems mechanism: every initialization leading to one solution lies arbitrarily close to initializations leading to different solutions, forming a fat fractal with structure at arbitrarily fine scales. Using a minimal neural network model, we analytically establish the classical route to riddling and derive a near-zero uncertainty exponent: even sharp increases in initialization precision yield only marginal gains in outcome predictability. In deep networks, intertwined training destinations and near-zero parameter-space exponents provide robust signatures consistent with riddling across tested architectures, optimizers and training protocols, offering a common dynamical account of otherwise disparate empirical observations. We further introduce functional uncertainty exponents, which extend these diagnostics of scale-insensitive irreproducibility from parameter space to the learned functions themselves. Together, our results support riddled basin geometry as a candidate organizing principle for irreproducibility in deep learning, with practical implications for optimization and the reliable deployment of artificial intelligence.