It is shown that, within a structured, single-textbook corpus, embedding model choice has significantly greater impact on RAG system reliability and performance than chunking strategy, and that open-source, evidence-driven architectures offer a transparent alternative for textbook-based clinical QA settings.
Abstract
Large language models (LLMs) are being investigated for clinical decision support tools, but can produce unreliable outputs, including hallucinated information, overgeneralized answers, and responses not clearly derived from authoritative medical sources. To ground responses in high-quality medical texts, researchers developed Retrieval-Augmented Generation (RAG). In this work, we show that, within a structured, single-textbook corpus, embedding model choice has significantly greater impact on RAG system reliability and performance than chunking strategy. The system compares twelve embedding models—general-purpose and biomedical-specific—under token-level and character-level chunking, prompting the LLM to answer from retrieved snippets. We evaluate retrieval effectiveness using rank-based metrics and generation quality using standard measures across 232 questions. General-purpose embeddings, particularly Jina-Embeddings-v3 and BGE-base-en-v1.5, outperformed biomedical-specific encoders in both retrieval and generation. Across both strategies, higher retrieval precision translated directly into improved lexical and semantic alignment with gold references. Under our operational definition requiring explicit evidence spans, 35.6% of answered questions lacked supporting evidence; our best configuration (BGE-base-en-v1.5, token-level) achieved the lowest rate at 26.1%—an evidence-grounding measure, not a guarantee of correctness. These findings highlight the central role of retrieval quality in clinical RAG systems, showing that open-source, evidence-driven architectures offer a transparent alternative for textbook-based clinical QA settings.
Objective: The rapid advancement of large language models (LLMs) in healthcare has opened new possibilities for AI-assisted clinical decision-making; however, safe and effective deployment remains challenging because of hallucination and domain misalignment. This study introduces DocTalk, a doctor-assistive question-an...
Asad Gohar, Md Ariful Islam Mozumder, Moon-Il Joo et al.· 0 citations
Large language models (LLMs) are increasingly explored for biomedical research and clinical medicine, yet their use remains limited by hallucinations, missing provenance, domain drift, privacy constraints, and uncertain regulatory pathways. Retrieval-augmented generation (RAG) offers a pragmatic alternative to full mod...
M. Enayati, Vishnu Priya, E. Prochaska et al.· The Scientist· 0 citations
MERGE-Med is a hybrid multi-expert architecture that consists of a mixture-of-experts (MoE) layer with three domain-specialized LLaMA-3 8B models for diagnostic reasoning, treatment protocol generation, and patient-friendly communication and a contrastive fine-tuning framework validated by an ensembled DeBERTa-v3-large...
D. Tejaswi, P. R. Reddy, K. Shailaja et al.· Conference Proceedings in Sc...· 0 citations