Loop Dropout: Regularizing Shared Updates in Looped Language Models
Looped language models separate computational depth from parameter count by repeatedly applying the same transformer block. Adapting these models requires a shared update that remains effective as hidden states evolve throughout the recurrent computation. Our empirical analysis reveals a pronounced late-loop bias in st...
Zi-Rui Zhu, Hai-Lun Xu, Xuan-Lei Zhao et al.
· 0 citations