The Shape of Wisdom: How Language Models Converge on an Answer

Abstract

When a language model answers a factual question, where does its internal state go? We asked four open models (Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B and Llama-3.2-1B) 2,256 factual questions whose answers recur, such as the country a person is a citizen of. At every layer, we recorded the state from which the answer is produced. Questions that receive the same answer converge on a shared region of state space, which we summarise by its centre, the answer centroid. The answer becomes readable within a single layer. In every model, the share of questions whose state lies nearest its own answer's centroid jumps at one layer, between 47% and 71% of the way through the network. The centroids follow the answer the model gives rather than the question: the centroid of a country learned from questions about people's citizenship also identifies that country in questions about towns and films. The model also uses the centroids. After the jump, moving the state of a wrongly answered question to the right answer's centroid makes the model give the right answer in 83%-97% of cases late in the network, against 2%-4% for a random move of the same size. Before the jump, moves of the same size almost never work. Near misses, in which the model answers wrongly but picks the right option in multiple choice, can be seen from the onset layer, where the right answer's centroid is already among the nearest to the state. This signal is about as informative as the right answer's rank in the model's final output, and more informative than the model's confidence in its reply.