EviRAG: Evidence-Guided Retrieval-Augmented Generation for Medical Vision-Language Models

Abstract

Retrieval-augmented generation (RAG) is widely adopted for radiology report generation with medical vision-language models, leveraging external reports as linguistic references. However, existing RAG methods rely primarily on dense embedding similarity, which may retrieve reports that are semantically related yet clinically inconsistent with respect to presence or laterality constraints. Such inconsistencies are often propagated into generation, resulting in contradictory or unsupported findings. We propose an evidence-guided retrieval-augmented framework EviRAG that decomposes retrieval into structured and unstructured alignment levels. First, we induce structured clinical triplets from both query and database cases through targeted visual interrogation, projecting images into a shared evidence space. Triplet-level alignment enforces explicit agreement over presence and laterality variables, yielding a clinically admissible candidate set via structural ranking. Within this constrained space, we perform semantic alignment in a shared multimodal embedding space to capture nuanced descriptive correspondence. The top-ranked reports and query image are jointly fed into a medical vision-language model for report generation. Comprehensive experiments on radiology report generation benchmarks show that EviRAG substantially reduces clinical inconsistencies compared to strong medical vision-language baselines. The source code is available at https://github.com/liamgu06/EviRAG.