Comparing FAQ-Based and Retrieval-Augmented Generation Chatbots for Academic Service Question Answering

Abstract

Academic service units need question-answering systems that respond quickly while remaining aligned with institutional regulations. This study provides a controlled, deployment-oriented comparison of a lightweight FAQ chatbot and a Retrieval-Augmented Generation (RAG) chatbot for academic-service question answering. The novelty lies in evaluating both prototypes against the same version-controlled official-document scope, the same 60-question dataset, category-level service cases, paired statistical tests, inter-rater reliability, and RAGAS retrieval-quality metrics. The questions covered registration, KRS/KHS, schedules, tuition payment, thesis or final project, and administrative letters. Three validators assessed accuracy, relevance, response effectiveness, and response time using official documents as the reference standard. RAG achieved 88.33% accuracy, a mean relevance score of 4.48/5, and 86.67% response effectiveness; FAQ achieved 68.33%, 3.46/5, and 65.00%, respectively. Paired tests confirmed significant advantages for RAG in accuracy, relevance, and effectiveness, while FAQ was significantly faster (0.62 s vs 2.41 s). RAGAS evaluation showed context precision of 0.86, context recall of 0.82, faithfulness of 0.89, and answer relevancy of 0.91. The findings support a cautious hybrid strategy: FAQ for stable repetitive requests and RAG for contextual, document-dependent questions, subject to broader testing with live queries and stronger retrieval baselines.