Information-Theoretic Limits on Graph Neural Network Compression

Abstract

Graph Neural Networks (GNNs) have achieved remarkable success in various domains, including node classification, graph classification, and link prediction. However, their large size and high computational cost pose significant challenges for deployment, particularly on resource-constrained devices. Graph Neural Network compression aims to reduce the model size and computational complexity while preserving performance. This work investigates the fundamental limits of GNN compression from an information-theoretic perspective. We analyze the redundancy inherent in GNN parameters and leverage concepts from source coding theory to guide the development of compression techniques. Our analysis reveals that significant compression is possible, but it's inherently limited by the information content captured by the GNN. We propose a framework for quantifying this limit and designing compression strategies that minimize information loss. The primary contribution is a rigorous, information-theoretic foundation for GNN compression, offering insights into the trade-offs between compression ratio and performance degradation. The key metrics used are the source coding efficiency (SE) and the mutual information (MI) between original and compressed representations. We demonstrate through theoretical analysis and a simplified illustrative example that achieving high compression ratios without considering these information-theoretic constraints can lead to substantial performance degradation.