Hard-Wiring the Induction Head: The Surprising Effectiveness of Copy Mechanisms in Data-Efficient Language Modelling
Large language models need several orders of magnitude more text than a child hears to reach a comparable level of linguistic skill. One reason is that a particularly useful capability, the induction head (a circuit that completes a pattern of the form [A] [B] ... [A] by copying [B]), is not built in but must emerge du...