It is shown that one-layer transformers with an argmax classification head behave identically to a one-nearest-neighbor classifier in the multiclass setting by leveraging the simplex encoding.
Abstract
We extend recent work establishing an equivalence between one-layer transformers and nearest-neighbor classifiers in the binary setting to the multiclass case. By leveraging the simplex encoding, we show that one-layer transformers with an argmax classification head behave identically to a one-nearest-neighbor classifier in the multiclass setting. This closes a gap left by prior work, whose multiclass result relied on a non-standard rounding-based approach rather than the typical argmax head used in practice.
Across CIFAR-10, CIFAR-100, and TinyImageNet with ResNet and vision transformer backbones, lifted training improves test accuracy by up to five percentage points over variants without lifting under a shared tuning protocol.
Robert Lampel, Timon Klein, Sebastian Sager· 0 citations
In this presentation, we introduce a distributionally robust Twin Support Vector Machine (TSVM) model designed to tackle bi-class and multi-class classification challenges under uncertainty. Our model leverages first and second-order information to enhance robustness. By solving two interconnected chance-constrained SV...
VAST (Veracity-Aware Semi-Supervised Training) outperforms the strongest graph-based SSL baselines at every operating point across three datasets, with statistically significant gains in 7 of 9 comparisons, while producing a deployable inductive classifier rather than requiring transductive graph inference.
We investigate a general-purpose layer for deep learning that, instead of compressing arbitrary-size training data into fixed-size weight matrices, stores a new pair of key-value representations for every data point during training, and retrieves and recombines these representations through an attention mechanism at in...
Maximilian Schlegel, Rajai Nasser, Seijin Kobayashi et al.· 0 citations
This paper studies what happens to the rest of the model when a class is forgotten, using a label-conditioned energy-based model (EBM) that assigns per-class energies, making the effect directly observable.
Syed Ali Ahmed, Syed Bilal Ahsan, Muhammad Zaigham Zaheer National University of Computer et al.· 0 citations
It is shown that adding a free affine skip connection leaves these function-space solutions unchanged but fundamentally improves the parameter-space landscape: every KKT point of the constrained problem becomes globally optimal, whereas suboptimal KKT points can occur without the skip connection.
Karolina Drabik, Ben Lewis, Antoni Puch et al.· 0 citations
Learn how Google DeepMind and GovTech Singapore uses lightweight on-device Gemma models to intercept conversational scams in private chats with zero cloud data egress.
