Functional Degeneracy in Neural Networks: Measurement and Pruning
This work quantifies functional degeneracy through the behavioral recovery rank, defined as the number of leading behavioral-Hessian eigendirections required to recover a trained model's performance, and finds that structural and magnitude pruning retain more degrees of freedom, even after the task is saturated.
M. Matveev, P. Esser, Ayush Bharadwaj et al.
· 0 citations