An Introduction to Stochastic Deep Learning

Abstract

Deep neural networks (DNNs) have achieved remarkable success in prediction, but their deterministic formulation makes many statistical inference tasks difficult. StoNet, short for stochastic neural network, addresses this limitation by reformulating a DNN as a probabilistic latent‐variable model, in which the outputs of selected hidden layers or units are treated as latent variables. The article describes the formulation of StoNet and its asymptotic equivalence to a conventional DNN, showing how StoNet can serve both as a deep learning model and as an analytical device for studying DNNs. It reviews training algorithms for StoNet and discusses its applications to statistical inference problems, including nonlinear sufficient dimension reduction, causal inference with observed and unobserved confounders, Granger‐causality learning for nonlinear time series, nonlinear variable selection, and prediction uncertainty quantification for large‐scale DNNs. The article also discusses two related architectures: kernel‐expanded StoNet and sublinearly structured DNNs, which can be viewed as structural or theoretical extensions of the StoNet framework. Under suitable architectural conditions, the former is immune to local traps in training and tends to outperform conventional DNNs in prediction; whereas the latter achieves feature‐learning consistency in over‐parameterized regimes when the training sample size is sufficiently large. Together, these developments show that StoNet broadens deep learning from prediction‐oriented function approximation to a richer framework for statistical inference.