Optimizing RAG-Based Academic Chatbot Performance Using Prompt Engineering
TL;DR
Findings indicate that clear prompt instructions can improve the reliability of RAG-based academic chatbot responses for academic information services and show the strongest improvement in faithfulness and context recall.
Abstract
Background: The increasing use of artificial intelligence in higher education has encouraged institutions to develop academic chatbots that provide faster access to official information. However, Retrieval-Augmented Generation (RAG)-based chatbots still require optimization to ensure accurate, relevant, and context-grounded responses.Aims: This study aims to optimize the performance of a RAG-based academic chatbot by applying zero-shot and few-shot prompt engineering strategies.Methods: A comparative experiment was conducted using 45 in-context academic questions for RAGAS-based quantitative evaluation, while 5 out-of-context questions were used as a qualitative robustness check. The system was developed using Python, LangChain, FAISS, OpenAI, and Streamlit, and evaluated using the RAGAS metrics: faithfulness, answer relevancy, context precision, and context recall.Results: The baseline system achieved an average RAGAS score of 0.8421. After prompt engineering was applied, zero-shot prompting achieved 0.8697, while few-shot prompting achieved 0.8565.Conclusion: Zero-shot prompting produced the best overall performance and showed the strongest improvement in faithfulness and context recall. These findings indicate that clear prompt instructions can improve the reliability of RAG-based academic chatbot responses for academic information services.