A variational approach to a finite-temperature continuous-spin perceptron trained on a Gaussian mixture is introduced and lower and upper minimax variational bounds for the limiting quenched pressure are derived.
Abstract
We introduce a variational approach to a finite-temperature continuous-spin perceptron trained on a Gaussian mixture. The model allows for a broad class of concave utilities and log-concave separable prior measures on the spins. By combining the interpolation method with log-concavity and concentration estimates, we derive lower and upper minimax variational bounds for the limiting quenched pressure. Remarkably, the two bounds differ only in the order of optimization of two variational parameters, while all remaining extrema are controlled by the concave--convex structure of the variational potential. Whenever the two optimizations commute, the two bounds match and identify the solution of the model. The same potential yields the fixed-point equations as stationarity conditions and provides a unified route to the computation of the ground-state energy, training loss, and generalization error.
Using a finite energy message-passing algorithm, it is demonstrated numerically that thermal noise enables effective generalization in the regime of constraint densities where both recovering the teacher and finding a zero temperature solution are computationally hard.
Enrico M. Malatesta, A. Passalacqua, Riccardo Zecchina· arXiv.org· 0 citations
An algorithm for composite optimization problems of the form min x f (x) + g(T x), where f is smooth and g may be non-smooth is proposed, which leverages the Moreau envelope to smooth the non-smooth component while adapting to problem geometry through linear minimization oracles.
A variant of stochastic gradient descent with initial regularization with initial regularization is analyzed and dimension-free upper bounds on its expected excess risk for the squared loss are derived.
It is shown that a symmetry fixes what variables are: a network layer is a sum over interchangeable units, so relabeling the units leaves it unchanged; given smoothness and the condition that a unit's gradient vanish at the origin, symmetry then enforces a universal leading form for the expansion about the near-zero we...
Zi-Yin Liu, Yi-Zhou Xu, Tomaso A. Poggio et al.· 0 citations
The results show that, for the simple network model, if the variance of the initialization distribution satisfies the theoretical optimal condition, then the corresponding network achieves lower final training loss and higher test accuracy than the conventional He-normal initialization.
H. Horii, S. Has· IEEE Transactions on Neural...· 0 citations
We study the problem of sampling from target distributions whose potentials are simultaneously non-smooth, subject to superlinear gradient growth, and non-convex. We introduce the Subgradient Tamed Unadjusted Langevin Algorithm (SG-TULA), a discretisation of the Langevin diffusion that operates directly on subgradients...
Iosif Lytras, Nikolaos Makras, S. Sabanis· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
