One-Layer Transformer Provably Learns Multiclass One-Nearest Neighbor in Context
It is shown that one-layer transformers with an argmax classification head behave identically to a one-nearest-neighbor classifier in the multiclass setting by leveraging the simplex encoding.
Skanda Athreya, Yu-Tong Wang
· 0 citations