Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals
In this paper, we argue that looped Transformers need their own residual connections to prevent performance degradation as the number of iterations grows. We observe that increasing loop iterations can reduce reasoning accuracy: noisy state updates overwrite correct intermediate deductions and even undo completed solut...
Peng-Xiang Li, Dilxat Muhtar, Di He et al.
· 0 citations