Xin-Yang Hu

Understanding In-context Learning of Addition via Activation Subspaces

A novel method is introduced that localizes the model's few-shot learning ability to only a few attention heads through dimensionality reduction and decomposition of the heads'output spaces, and derives a mathematical identity relating the''aggregator''and''extractor''subspaces of attention heads, allowing us to track...