Intent Classification Under Label Noise: A Comparative Analysis of Fine-Tuned Transformers and Large Language Models

Abstract

Label noise, which is frequently encountered in real-world data, is a critical problem that can directly degrade model performance. In this study, we systematically investigated the impact of label noise on intent classification by comparing the in-context learning (ICL) approach of large language models (LLMs) with fine-tuned transformer models. In the experiments conducted on the Banking77 dataset, we evaluated four LLMs and two transformer models under three different noise types and three different noise levels. We also tested the LLMs with different prompting strategies to examine the effect of taking precautions against potential noise on different LLMs. Our findings show that strong LLMs experience less than 2 percent loss in accuracy and F1 score even under the heaviest noise conditions, whereas in fine-tuned transformer models and relatively weaker LLMs, the drop can reach the 15-20 percent range.