Adaptive self-organized criticality in deep neural networks
TL;DR
This work shows that the global dynamical state of a deep neural network can be autonomously regulated by purely local homeostatic plasticity, and demonstrates how adaptive self-organization can be implemented in deep neural networks and how local plasticity can control their collective dynamical operating point.
Abstract
Deep neural networks are high-dimensional dynamical systems whose function depends on the stable propagation of activity and perturbations across many layers. Maintaining suitable dynamical regimes may therefore be essential for robust learning and for preventing dynamical instabilities during training. Here, we show that the global dynamical state of a deep neural network can be autonomously regulated by purely local homeostatic plasticity. Neuronal activity is inferred from responses across inputs, and individual synapses are strengthened or weakened using only the activity of their postsynaptic neuron. Without measuring any global network property, this rule drives networks from both subcritical and supercritical initial conditions toward a common critical state, characterized by conserved activity propagation and a vanishing largest finite-time Lyapunov exponent. When combined with gradient-based learning, homeostatic adaptation counteracts the training-induced drift toward supercritical dynamics, while revealing a competition between dynamical regulation and task optimization. Our results demonstrate how adaptive self-organization can be implemented in deep neural networks and how local plasticity can control their collective dynamical operating point.