M. Mozer

recirculation

We describe an inference-time architectural enhancement for off-the-shelf foundation models that systematically reduces perplexity and boosts accuracy across generation and reasoning tasks. Our approach incurs essentially no additional latency during generation, though it requires serial processing in the prefill phase...

Neural Network Online Training With Sensitivity to Multiscale Temporal Structure

The Topological Trouble With Transformers

It is argued that temporally extended cognition requires refocusing from explicit thought traces to implicit activation dynamics via recurrent architectures via recurrent architectures, and a taxonomy of recurrent and continuous-thought transformer architectures is introduced.