Label space reduction for transductive zero-shot classification with large language models

Abstract

Zero-shot classification enables models to assign labels to unseen classes without task-specific training and has become increasingly effective with the advent of Large Language Models (LLMs). However, existing zero-shot classification approaches typically present all candidate labels as a flat, unstructured list within the prompt. As the number of classes grows, this formulation exacerbates well-known limitations of LLMs, including attention dilution and positional bias, ultimately degrading classification performance. We present Label Space Reduction (LSR), a test-time training method for transductive zero-shot LLM classification. LSR uses LLM-generated pseudo-labels to iteratively refine the classification label space by systematically ranking and reducing candidate classes, enabling the model to concentrate on the most relevant options. By leveraging unlabeled data with the statistical learning capabilities of data-driven models, LSR dynamically optimizes the label space representation at test time. Our experiments across seven benchmarks demonstrate that LSR improves macro-F1 scores by an average of 7.0% (up to 14.2%) with Llama-3.1-70B and 3.3% (up to 11.1%) with Claude-3.5-Sonnet compared to standard zero-shot classification baselines. To reduce the computational overhead of LSR, which requires an additional LLM call at each iteration, we propose distilling the model into a probabilistic classifier, enabling lightweight deployment without repeated LLM calls.