Mind the Approximation: Fisher-Weighted SVD Compression for ViTs
FACTS, a structured Fisher Approximation tailored to Compressing ViTs with Fisher-weighted SVD, which enforces token-local aggregation while preserving within-token activation-gradient dependence, and introduces a fast Constrained Rank Search (CoRS), that optimizes layer-wise rank allocation while adhering to a fixed f...
Moritz Thoma, Maximilian Groezinger, Maximilian Forstenhäusler et al.
· 0 citations