Skip to content
Conference

Structured Hallucinations in Small Language Models: Repeatability, Attention Signatures, and Context Contagion

Oct 2026 · International Conference on Information and Software Technologies · pp. 17-25 · 0 citations · 26 references

Abstract

In this paper, we present an empirical analysis of hallucination behavior in small language models (1.5B–7B parameters), addressing the question of whether such failures are stochastic or structured. Our analysis of 4,500 responses from three instruction-tuned models, evaluated on TruthfulQA and HaluEval, reveals that hallucinations exhibit highly consistent, non-random patterns. First, 92.85% of question–model pairs that hallucinate do so consistently across repeated runs (χ2 =4316.4, p < 0.001), indicating highly repeatable failure modes tied to specific queries. Second, layer-wise attention entropy reveals a consistent mechanistic signature: hallucinated responses exhibit significantly higher early-layer entropy in two distinct architectures (LLaMA: t = 13.305; Mistral: t = 8.976; p < 0.001). Third, the introduction of false contextual information increases the hallucination rate by 1.69× (p = 0.0003), with modelspecific amplification effects reaching up to 3.06×. Furthermore, hallucinated responses exhibit model-specific lexical signatures. A TF-IDF classifier achieves > 86.75% accuracy in identifying the source model, substantially outperforming semantic embedding-based methods (63.11%). Together, these findings demonstrate that hallucinations in small language models are structured and reproducible, underscoring the need for targeted mitigation strategies rather than reliance on scaling alone.

View source

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.