Hao-Tian Gu

In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners

In-context learning (ICL) enables a pretrained model to infer a task from demonstrations without updating its parameters. While much of the existing theory focuses on linear target functions, in this paper we study nonlinear cases by comparing two one-layer attention architectures on the same family of single-index tas...