ActTail: Global Activation Sparsity in Large Language Models
ActTail is proposed, a TopK magnitude-based activation sparsity method with global activation sparsity allocation grounded in Heavy-Tailed Self-Regularization (HT-SR) theory that improves both perplexity and downstream task performance at high sparsity compared to uniform allocation.