How Does Machine Learning Manage Complexity?

Abstract

We provide a computational complexity lens to understand the power of machine learning models, particularly their ability to model complex systems. Machine learning models are often trained on data drawn from sampleable or more complex distributions, a far wider range of distributions than just computable ones. By focusing on computable distributions, machine learning models can better manage complexity via probability. We abstract away from specific learning mechanisms, modeling machine learning as producing P/poly-computable distributions with polynomially-bounded max-entropy. We illustrate how learning computable distributions models complexity by showing that if a machine learning model produces a distribution $\mu$ that models the distribution generated by a cryptographic pseudorandom generator at least as well as the uniform distribution does, then $\mu$ must be close to uniform. We view training as compression that removes the randomness from the data, keeping only its structure, and inference as adding randomness back to generate new outputs. A computationally bounded model cannot distinguish pseudorandomness from true randomness, so training removes both, treating anything it cannot efficiently predict as noise. The same holds at inference, where the randomness added back can be, and in practice is, pseudorandom. In the appendix, we make the connection to compression precise and, using a characterization of pseudoentropy by Vadhan and Zheng, show that the best loss achievable by an efficient model is the next-bit pseudoentropy of the data.