ARLoop: Fully-Open Pre-trained Looped Transformers with Attention Residuals
Looped transformers increase effective depth by repeatedly applying a shared set of transformer layers, offering a parameter-efficient way to scale computation. However, standard residual connections aggregate information across recurrent depth into a single stream, providing limited control over which intermediate rep...