Benchmarking Classical, Transformer-Based, and RAG Approaches with LLMs for Automated Bug Triage

Abstract

: Automatic bug triage is a critical software maintenance task that reduces the high human resource cost of manual developer assignment. In this paper we evaluate various machine learning approaches on Eclipse dataset. We compare classical machine learning approaches like TF IDF + XGBoost, LogReg, NaiveBayes and modern transformer based architectures including: Bert, Nomic, Mpnet, Jina, BGE, E5, CodeLlama, Llama-3.1-8B-Instruct, ChatGPT few-shot prompting. We compare various approaches including using transformer based architectures with frozen weights as embedding with a fine-tuned neural network classification head (MLP). For maximal performance we selected the best performing model E5 (47.63% accuracy) and fully fine-tuned it and achieved 72.49% top-1 accuracy which is the best performing deep learning based approach to the best of our knowledge slightly under performing our best approach: TF IDF+XGBoost 76.47%. Furthermore we also experienced with adding the embedding vectors of some of the aforementioned transformer-based models to the best performing TF IDF+XGBoost models but we experienced slightly worse performance than with only TF IDF+XGBoost. We also experimented with Retrieval Augmented Generation (RAG) and achieved 36.81% top-1 accuracy. The RAG results were primarily influenced by the retrieval rather than the generator.