Evaluating RAG Configurations for Clinical Information Extraction from EHR Notes: Aged Care Case Study

Abstract

Clinical information extraction from unstructured electronic health records is important for supporting clinical decision making and healthcare research. However, large language models can struggle to accurately extract domain-specific information without effective adaptation. Retrieval-augmented generation offers a promising approach, but the optimal configuration for clinical information extraction remains unclear. This study evaluated different retrieval-augmented generation configurations for extracting frailty index items, malnutrition risk factors, and agitation in dementia from clinical notes collected from 40 Australian residential aged care facilities. We compared general and biomedical embedding models, LangChain and LlamaIndex retrieval frameworks, top-k values of 1, 5, 10, 15, and 20, and one to five few-shot examples using Llama-3.1-8B-Instruct. Performance was evaluated using accuracy, precision, recall, and F1 score. The best results were achieved using LangChain with the BAAI/bge-m3 embedding model, three-shot prompting, and a top-k value of five. This configuration achieved F1 scores of 97.08% for frailty index extraction, 97.14% for malnutrition risk factor extraction, and 90.52% for agitation in dementia extraction. These findings provide empirical evidence on how retrieval, embedding, and prompting choices influence clinical information extraction performance. The study offers practical guidance for researchers and healthcare informatics practitioners seeking to optimise retrieval-augmented large language model systems for extracting clinically meaningful information from unstructured aged care records.