No Such Emoji: A Controlled Benchmark and Activation-Patching Analysis of Vocabulary-Gap Confabulation

Abstract

Asked for the seahorse emoji, language models often supply a wrong one. Unicode has none, so the correct answer is a refusal. Using the Unicode registry as an exact oracle, we build 180 probe items (123 in an absent-emoji core) and 174 matched controls, scored under a five-category taxonomy. On Qwen2.5-Instruct at 0.5B, 1.5B, and 7B, undisclaimed-emission rates on the core are 61.8%, 56.9%, and 85.4% (84% to 89% at every size on the two templates the smaller models answer). At 7B the model emits an emoji about as often when no correct one exists as when one does (88.6% vs. 92.4%). The smaller models never say an emoji does not exist; at 7B such statements reach 5.7% of core probes. In a patching pilot, swapping one late residual vector at the emission position between a confabulating and a correct run makes each emit the other's first answer token in every non-degenerate pair (all 10 at 1.5B, all 12 at 7B). The late vector thus controls the first answer token; whether the model represents the concept and fails only at the output, as a logit-lens blog analysis argued, stays open. We release all data and code. Disclosure: Claude (Anthropic), a large language model, assisted this work under my direction: it built the benchmark construction and scoring software, ran the model probes and the activation-patching experiment, checked the bibliographic references against their sources, and assisted with the writing throughout. The text is my own, and I take full responsibility for every sentence and number in it. Code and data: https://github.com/Tyorden/vocab-gap-confabulation. Its release v1.0 (doi:10.5281/zenodo.22054976) predates the 7B runs, the patching baselines and the fictitious-referent control; the repository holds them. The two archives deposited with this preprint hold every script and results file the paper uses, including the Appendix D analyses.