Where Multi-Step Reasoning Breaks: Measuring Error Propagation and Structured Scaffolds in Small Language Models

Abstract

Chain-of-thought prompting lets a language model write out its intermediate reasoning, and a wrong intermediate claim can carry through to a wrong final answer. This preprint measures that error propagation directly for Qwen2.5-1.5B, Qwen2.5-3B and Llama-3.2-3B Instruct, run locally in 4-bit. On ProofWriter, every ground claim a model asserts is checked against the theory's full derivation closure. On GSM8K, every stated calculation is checked for arithmetic and against the intermediate values of the reference solution. The paper then asks whether structured scaffolds (explicit step decomposition, step-wise decoding with a self-verifier or an oracle verifier, and self-consistency) reduce step errors, their propagation, or both. Chains that contain a step error do end wrong more often, and the association survives adjustment for problem depth. Most wrong answers, though, contain no checkable step error. On ProofWriter they fail at the final inference about the queried statement, never state it, or give no answer. On GSM8K they compute the wrong quantity with correct arithmetic (0.628, 0.723 and 0.680 of wrong chain-of-thought chains for the three models). An oracle verifier that removes checkable step errors by construction does not beat chain-of-thought on either benchmark for either Qwen model, and a self-verifier does no better. Across eight model and benchmark cells, including FOLIO, structured decomposition is never measurably better than chain-of-thought, and self-consistency gives the only consistent gains, which are small. On GSM8K the oracle result holds under a second prompt wording. For models of this size, planning and the final inference are the bottleneck, and intermediate slips account for a minority of wrong answers. Preprint, not peer-reviewed. All models ran on a personal laptop and all datasets are public.