Preprint
Aug 2026
How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention
This paper provides a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention, and extends the analysis to fused multi-head LoRA and joint query/key updates, exposing the effects of rank sharing and query/key factorization constraints.
G. Planes
· 0 citations