Rupa Laalasa Oruganti

Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders

The quadratic scaling of Transformer self-attention has driven the adoption of sub-quadratic Selective State Space Models (SSMs) like Mamba, which compress past context into a fixed-size recurrent hidden state. This strict informational bottleneck raises a foundational question for mechanistic interpretability: do SSMs...