CADSA: Content-Aware Dynamic Sparse Attention for Lightweight and Energy-Efficient Edge AI with Sub-Quadratic Complexity
CADSA is a content-aware dynamic sparse attention framework that lowers the quadratic complexity of self-attention to sub-quadratic for lightweight edge AI deployment, and fine-tuning strategy with gradient masking decreases the training memory overhead by 60%, allowing for practical on-device fine-tuning.
Abdullah Ghanim Jaber, Abeer Ahmed Ali, Ibrahim Alshammari et al.
· Iraqi Journal for Computers... · 0 citations