Jul 2026
DeepLoop: Depth Scaling for Looped Transformers
The results show that stable recurrent depth requires residual scaling rules that account for parameter visits, not only nominal layer count, and DeepLoop is neutral when no physical block is revisited and improves validation loss and downstream accuracy once recurrent depth is activated.
Shu-zhen Li, Yi-Fan Zhang, Jiacheng Guo et al.
· arXiv.org · 2 citations