GuidelineGuard: An Agentic Retrieval-Augmented Generation Framework with Sentence-Level Citation Auditing for Guideline-Grounded Question Answering
TL;DR
GuidelinesGuard, a modular multi-agent retrieval-augmented generation pipeline in which a separate Auditor verifies claim–sentence support before a recommendation is surfaced, is introduced as a selective claim–evidence verification mechanism.
Abstract
Background: Large language models (LLMs) can produce clinically plausible recommendations that are not adequately supported by authoritative evidence. Objectives: We introduce GuidelineGuard, a modular multi-agent retrieval-augmented generation pipeline in which a separate Auditor verifies claim–sentence support before a recommendation is surfaced. Methods: The original evaluation used a 73-sentence guideline corpus and GG-Bench-60, with replication across three open-weight backbones. In response to reviewer concerns about benchmark size and selective evaluation, we added a source-traceable GG-Bench-200 stress test and the complete 500-case held-out PQA-L test split of PubMedQA. The revision experiments compare single-pass RAG, a paired multi-agent no-Auditor ablation, and GuidelineGuard; the paired runner is designed to share the Planner–Retriever–Clinician draft so that the Auditor is the only intervention. Checkpoint verification confirmed an identical observable pre-audit state for all 200 GG-Bench cases and 496/500 PubMedQA cases; four PubMedQA cases were regenerated after quota-interrupted resumption and were correct commitments in both arms. Because the originally used hosted Llama endpoints became unavailable after the initial experiments, the expanded runs use openai/gpt-oss-20b for generation and openai/gpt-oss-120b for the Auditor. Results: On GG-Bench-200, single-pass RAG achieved 0.970 operational accuracy, while the no-Auditor and GuidelineGuard arms achieved 0.955 and 0.925, respectively. GuidelineGuard committed on 186/200 cases (coverage 0.930) and was correct on 185/186 commitments (selective accuracy 0.995); all 186 commitments cited at least one gold evidence identifier. Relative to the paired no-Auditor arm, the gate rejected six otherwise-correct commitments and no incorrect commitment. On PubMedQA-500, single-pass RAG achieved 0.644 operational accuracy at 0.950 coverage, the no-Auditor arm 0.638 at 0.896 coverage, and GuidelineGuard 0.550 at 0.736 coverage. Selective accuracy increased across those operating points from 0.678 to 0.712 to 0.747. Within the 496 PubMedQA cases with verified-identical observable pre-audit state, the gate rejected 36 incorrect and 45 correct pre-audit commitments, demonstrating both error enrichment and a substantial false-rejection cost. Conclusions: The expanded results support GuidelineGuard as a selective claim–evidence verification mechanism, not as a universally more accurate generator. Its value is the explicit, auditable coverage–risk trade-off; the appropriate verification threshold is task- and cost-dependent and requires prospective clinical validation.