This work introduces Soft Latent Thinking, a method that replaces the LM head during reasoning with a lightweight projector, enabling autoregressive rollout in embedding space where reasoning steps remain continuous rather than tokenized.
Abstract
Large language models decode by projecting hidden states through a large vocabulary head at every step. This operation is computationally costly and forces all reasoning to be expressed in discrete tokens. We introduce Soft Latent Thinking, a method that replaces the LM head during reasoning with a lightweight projector, enabling autoregressive rollout in embedding space where reasoning steps remain continuous rather than tokenized. Experiments on DeepSeek-Qwen-1.5B and LLaMA-3.2-3B show that Soft Latent Thinking consistently improves pass@k across all k while reducing per-step compute during chain-of-thought. Our method achieves the highest pass@32 among all soft-thinking approaches, demonstrating that effective reasoning can be carried out in continuous space without discrete token generation.
Latent Recurrent Thoughts substantially outperforms prior frozen-decoder continuous-space reasoning methods under an identical decoder, prompt, data, and training budget, and outperforms non-thinking-mode chain-of-thought prompting on the same backbone at a small fraction of its inference compute.
REST (REpresentation-Supervised Thoughts), a training objective that turns four properties of a valid thought representation into differentiable losses added to CE, instantiate in latent single-agent and multi-agent systems, without architectural changes or added parameters at inference.
Autoregressive Thought Flow is introduced, which models the next continuous thought as a multimodal distribution, and suggests that continuous reasoning is more effective when multiple possible next thoughts remain available rather than being collapsed into a single prediction.
Yang Li, Yi Wang, Shinda Huang et al.· 0 citations
Activation steering has become a widely used approach for controlling language models during explicit chain-of-thought (CoT) reasoning, motivating its extension to latent CoT. However, we find that steering continuous thoughts produces substantially weaker effects on subsequent language generation than steering explici...
It is demonstrated that valence-related activation patterns leave hidden traces that predictably govern later choices, even when all visible tokens are identical across conditions, and whether these traces are accompanied by any subjective experience relevant to model welfare remains unclear.
This work applies Top-K Sparse Autoencoders to the intermediate representations of DeepSeek-R1-Distill-Qwen-7B and examines the model's divergent behaviors across math-solving tasks of three distinct difficulty levels, identifying a clear distinction in how the model functions under two reasoning modes.
Bo Cheng, Qiaolin Lu, Yi Chang et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduAug 27, 2026
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
