A decoupling in how a transformer organizes causal knowledge is reported: slot-by-type structure induced by type-level supervision organizes routing, yet remains functionally decoupled from answer readout.
Abstract
When a language model answers an interventional question, the computation it must perform depends on the type of evidence the query requires. We report a decoupling in how a transformer organizes causal knowledge: slot-by-type structure induced by type-level supervision organizes routing, yet remains functionally decoupled from answer readout. We establish this with a typed mechanism library -- discrete mechanism slots partitioned by evidence type, auditable at the state level -- on a causal-world benchmark with exact interventional ground truth, under a frozen protocol, at two scales (22.6M and 125M). Four preregistered findings. (i) Origin. Slot-by-type organization is induced by type-level supervision: absent in architecturally identical unsupervised controls, not buyable by content-free gating labels, and statistically attributable to the supervision signal, replicating at 125M under a powered preregistered protocol (all nine cells passed). (ii) Boundary. The induced structure is a typed routing index with a sharp routing/readout boundary: slot codes scaffold routing but do not drive answer readout ($|\Delta\hat{y}| \le 3.4\times10^{-6}$, zero collateral, three seeds, stable across a 5.6x scale window) -- we therefore make no behavioral-editability claim. (iii) Cost. The structure is free: LM quality matches a parameter-matched monolith within 0.0082 nats. (iv) Trust. The library state is exactly local under edit and bit-exactly revertible -- 250 single-edit and 1,000 stacked reverts per seed, zero failures. We further find that the unsupervised null itself moves with scale, so comparisons reusing a null calibrated at one scale may be confounded at another. Every claim is tied to a preregistered, machine-checkable criterion archived before the data it governs; the full audit trail, including one criterion we failed and how the frozen protocol handled it, is released as an appendix.
A two-level internal readout for mixture-of-experts reasoning that makes latent process state readable, while routing makes it deployable and actionable and improves single-branch selection and plain majority voting over completed candidate sets.
Kang Chen, Si-Han Zhao, Yi-Xin Cao et al.· 0 citations
A controlled proof of concept that information about a verified, causally relevant internal state can be preserved in generated text even when the answer is unchanged is provided.
These results support an economical reusable causal interface within the tested operation banks, while keeping the claim explicitly conditional on the candidate architectures, interventions, and held-out futures.
Siyuan Ma, Yiqin Luo, Zhangji et al.· arXiv.org· 4 citations
The complete pipeline -- detect, localize, and release -- is submitted to a fully preregistered stress test on a 25.7M transformer trained on causal-evidence discrimination, where a known suppression phenomenon (latent causal structure present but behaviorally unused) has previously been documented.
This work develops an algebraic formalization that derives expressivity directly from the model's implemented dynamics, and obtains an expressivity hierarchy governed by the attention type under specific numerical semantics.
Franz Nowak, Ryan Cotterell, Reda Boumasmoud· arXiv.org· 0 citations
PASK (Parser-Aware Structural KV Persistence), which turns parser-derived structure into layer-group-specific KV persistence decisions by using task-error sensitivity to set minimum protection floors and attention-output distortion to allocate residual KV capacity.
Lin Wu, Xin-Rong Chen· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.