Removable and Insertable Mechanisms for Small Language Models

Abstract

Can a small language model benefit from simple, hand-written tools for tasks such as copying a pattern or tracking brackets? We pair these tools with a standard transformer and test whether the model uses them by removing, swapping, or adding them. On structured synthetic tasks, explicit modules outperform tested generic banks on target behaviors at smaller approximate budgets. A lightweight copy-and-bracket bank improves targeted Python prediction and beats tested wrong, misaligned, and generic controls, but larger character-level banks lose to generic and approximately budget-matched transformers on real text. Results therefore distinguish targeted utility from overall language-model quality. Learned-module transfer is protocol-dependent. An initial attempt is compromised by next-token misalignment; a corrected fixed-mixture protocol finds transfer advantages on held-out domains, but a longer-adaptation protocol fails its frozen final multi-domain margin. Early advantages do not establish robust final superiority. Neural-to-explicit externalization succeeds in a forced one-head retrieval task: causal ablation and donor patching identify a necessary route, and a post-hoc lookup module restores behavior (7/9 frozen criteria). A corpus-derived retrieval follow-up fails the causal tests and yields a larger replacement; its final script summary crashes, so criteria are reconstructed from printed values. The positive result is a controlled existence proof for a known, localized computation not automatic circuit discovery, general externalization, universal portability, or competitive open-ended generation.