Evaluating Incremental Context Strategy in RAG-Based Chatbots for Hospital Document Retrieval: A Trade-off Between Quality and Cost

Abstract

Applications in the healthcare field where access to contextualized information anchored in reliable documents has been crucial have increasingly driven the adoption of Retrieval-Augmented Generation (RAG) systems. In this context, this work presents the development and evaluation of an RAG-based chatbot for retrieving normative documents from a university hospital. Traditional RAG approaches typically rely on fixed-size contexts, which may lead to excessive token consumption and degradation in response quality due to the inclusion of irrelevant information. Building upon a previously optimized retrieval pipeline based on the intfloat/multilingual-e5-small embedding model and Reciprocal Rank Fusion (RRF), this study investigates a novel incremental context strategy that progressively expands the input to the language model's during response generation. Experiments were conducted on a hybrid dataset of 192 question-answer pairs, combining synthetic queries and expert-generated questions. Four large language models (Gemini Flash 2.5, GPT-4.1-mini, Sabiá 3.1, and DeepSeek 3.2) were evaluated under fixed and incremental context settings, using BERTScore, LLM-as-a-Judge metrics, and token consumption as evaluation criteria. The results show that the incremental approach significantly reduces token usage, achieving up to 88.68% savings, while maintaining or improving semantic quality in most cases. These findings demonstrate that incremental context construction is an effective strategy for improving efficiency and response quality in RAG-based chatbots, while mitigating context dilution effects. The proposed approach is model-agnostic and readily applicable to real-world scenarios, particularly in resource-sensitive and high-stakes domains such as healthcare.