Pre-carved Niches: The Formation Dynamics of Modular Task Partitions in Early LLM Training
This work tracks formation step by step of a Pythia-410M model from scratch and runs attribution patching at every step, alongside probes for gradient norms, effective updates, weight norms, and first-order loss decomposition across 14 tasks in four cognitive domains, confirming the hypothesis that modularity tracks learning.