Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers
Looped transformers compute by applying one block repeatedly, so they can run for more loops than they were trained with. Recurrent reasoners are known to overthink: iterating past the trained depth can turn a correct answer into a wrong one. Such a late error can reflect failed computation, loss of retention, or a sta...
Pei-Long Zhou, Hong-Tao Zhang, Yu-Long Huang et al.
· 0 citations