A Comparative Study on LLM-enabled HPC User Support
TL;DR
A comparative study of eight candidate methods by combining the base model, SFT, RAG, and ICL using a ticket database of 60,000 question-answer pairs from Texas Advanced Computing Center suggests that combining ICL, RAG, and a powerful base model is a promising solution for LLM-enabled user support in computing centers.
Abstract
Providing accurate and efficient user support with the emerging large language model (LLM) techniques for high-performance computing (HPC) centers is extremely challenging. The major obstacle lies in the integration of specialized knowledge that is not available in the base LLMs. Supervised Fine-tuning (SFT), Retrieval Augmented Generation (RAG), and In-Context Learning (ICL), are the common techniques to empower LLMs with domain knowledge. However, there is a lack of comprehensive understanding of their question-answering capabilities in the HPC domain. In this work, we present a comparative study of eight candidate methods by combining the base model (Llama-3.1-8B-Instruct), SFT, RAG, and ICL using a ticket database of 60,000 question-answer pairs from Texas Advanced Computing Center (TACC). We evaluate these methods with five metrics: correctness, coherence, relevance, fluency, and cosine similarity, using both human and LLMs as judges. Our results show that ICL+RAG with the base model achieves the highest correctness in general, and ICL with the base model outperforms others for debugging and testing questions. The results suggest that combining ICL, RAG, and a powerful base model is a promising solution for LLM-enabled user support in computing centers.