Implementation of Retrieval-Augmented Generation (RAG) for Automatic Fact-Checking of Health Claims

Abstract

The rapid growth of digital platforms has accelerated public access to online health information, but it has also become a primary channel for the spread of health misinformation. Large Language Models (LLMs) have been widely adopted to address this problem through pure generative approaches; however, they frequently produce convincing yet inaccurate information, a phenomenon known as hallucination. This study implements a Retrieval-Augmentation Generation approach for automatic fact-checking of health claims using the PubHealth dataset, and compares its performance against a pure generative (Non-RAG) model. The proposed pipeline combines a FAISS-based semantic retriever, CrossEncoder re-ranking, and a FLAN-T5 generator to classify each claim as Fact or Myth while supplying relevant supporting evidence, and is integrated into a Gradio-based interface for claim verification and dataset expansion. The models were evaluated using accuracy, precision, recall, F1-Score, Cohen’s Kappa, faithfulness, context precision, and answer relevancy across four scenarios of question and claim normalization. The experimental results show that the RAG model outperformed the Non-RAG model on the classification metrics, with the best configuration achieving an F1-score of 0.7823, an accuracy of 0.7857, and a Cohen's Kappa of 0.5532, whereas the Non-RAG model suffered from extreme bias toward the Fact label and failed to detect the Myth label, reaching an accuracy of only 0.4211. Claim normalization was found to substantially improve classification accuracy, while the low faithfulness scores of the RAG model reflect a limitation of the metric on short binary answers rather than hallucination. These findings confirm that the retrieval and augmentation stages contribute significantly to producing more accurate, transparent, and evidence-based verification of health claims.