Exact capacity of the wide hidden layer treelike neural networks with generic activations.

Abstract

Recent progress in studying treelike committee machines (TCM) neural networks (NN) in Stojnic (2023c,g) showed that the Random Duality Theory (RDT) and its a partially lifted(pl RDT) variant are powerful tools that can be used for very precise networks capacity analysis. The initial considerations from Stojnic (2023c,g) regarding sign activations were extended to more general activations in Stojnic (2024), where particularly elegant results were obtained for any even number of quadratically and ReLU activated hidden layer neurons, d. While the results of Stojnic (2024) are applicable to any activation type in principle, a significant amount of numerical work is often required to make them practically usable. In this work, we examine wide hidden layer networks and uncover that certain aspects of such difficulties miraculously disappear. Specifically, we employ recently developed fully lifted (fl) RDT to characterize the capacity of wide (d → ∞) TCM nets. We obtain explicit, closed-form capacity characterizations for a generic class of hidden layer activations. While the utilized approach substantially reduces the necessary numerical evaluations, the ultimate success still requires a significant amount of residual numerical analysis. To determine the concrete capacity values, we analyzed four prominent activation examples: ReLU, quadratic, erf, and tanh. After successfully conducting all the residual numerical work for all of them, we have found that the lifting mechanism exhibits remarkably rapid convergence. Relative improvements do not exceed 0.01%-0.1% by the fourth level of lifting. As a convenient bonus, we also discover that the capacity characterizations obtained at the first and second lifting levels precisely match those derived through statistical physics replica theory methods in Zavatone-Veth and Pehlevan (2021) for generic activations and in Baldassi et al. (2019) for ReLU activations.