Beyond Lexical Similarity: Evaluating Faithfulness in LLM-Based Medical Question Reformulation
TL;DR
This work introduces MedFaith-F1, a category-level faithfulness metric over four clinically salient categories: clinical problems, medications, procedures, and follow-up intent, and proposes a hybrid Evidence and Knowledge-Grounded Retrieval-Augmented Generation ( EKG-RAG), an evidence and knowledge-grounded framework combining hybrid retrieval over PubMed and MedlinePlus resources with UMLS-aligned ontology grounding.
Abstract
Medical query rewriting transforms verbose consumer health questions into concise clinical queries, a critical step in health information retrieval. Large language models (LLMs) perform well on this task by standard metrics, yet high ROUGE or BERTScore does not guarantee preservation of clinical content. To address this issue, we introduce MedFaith-F1 , a category-level faithfulness metric over four clinically salient categories: clinical problems, medications, procedures, and follow-up intent. We further propose a hybrid Evidence and Knowledge-Grounded Retrieval-Augmented Generation ( EKG-RAG ), an evidence and knowledge-grounded framework combining hybrid retrieval over PubMed and MedlinePlus resources with UMLS (Unified Medical Language System)-aligned ontology grounding. Evaluating LLMs LLaMA-3 and Qwen2.5 across zero-shot, few-shot, and QLoRA settings on MeQSum and medical question-pair (MQP) datasets revealed that base models exhibit category-level faith-fulness failure rates (CHR) exceeding 40%, invisible to standard metrics, while EKG-RAG with QLoRA reduces CHR to 26.75%, achieving MedFaith-F1 of 0.73. Our findings call for faithfulness-aware evaluation in clinical query rewriting, and MedFaith-F1 provides a reproducible step in that direction.