Deep Neural Network Compression via Data-Driven Low-Rank Singular Value Decomposition
TL;DR
This paper proposes a novel compression method that uses Singular Value Decomposition (SVD) informed by data examples that achieves substantial reductions in model size and computation with minimal impact on accuracy and demonstrates its applicability in compressing deep neural networks to a fraction of their original size.
Abstract
Deep neural network compression focuses on reducing the number of parameters and computational complexity of neural networks, enabling their use in resource-constrained environments where smaller networks, faster response times, and lower energy consumption are critical. In this paper, we propose a novel compression method that uses Singular Value Decomposition (SVD) informed by data examples. Unlike existing SVD-based approaches that operate solely on network weights, our method leverages input data to more effectively preserve essential information during compression. This data-driven approach significantly enhances performance compared to state-of-the-art techniques. Through experiments on standard benchmark datasets, we show that our method achieves substantial reductions in model size and computation with minimal impact on accuracy. Furthermore, we demonstrate its applicability in compressing deep neural networks to a fraction of their original size, achieving competitive accuracy of networks with a compression ratio of less than 3× with no fine-tuning, and at most 25 epochs of fine-tuning for higher compression ratios.