GISum: Gibbs Inspired Summarization Architecture for Traceable Medical Summarization

Abstract

Document summarization has been significantly improved by large language models, still a lot of challenges exist, while dealing with medical documents. Hallucination is one of them, producing confident statements which are actually not factual. These LLMs don't indicate the origin of the information which is also a major concern in clinical environments. The TRACSUM benchmark is introduced to tackle this issue by asking models to generate summaries that are directly linked to specific sentences in the source data. Current methods achieve this by fine-tuning large models-but these systems often work like black boxes. In this work, we design GISum: Gibbs Inspired Summarization architecture for traceable summarization pipeline. Our proposed framework works in three stages: i)First, a reciprocal rank fusion(RRF) to gather relevant evidence by combining BM25, an MS-Marco Cross-Encoder, and PubMed-BERT; ii) second, a medical Gibbs sampler to evaluate and filter sentences based on clinical relevance, semantic alignment; and iii) lastly, in the third stage, an NLI-based gating step uses RoBERTa-Large-MNLI to check every claim generated by LLaMA 3.1, filtering out anything that can't be logically traced back to the source. Our proposed framework achieves a citation recall of 78.88% On the TRACSUM benchmark outperforming the LLaMA 3.1(8B) baseline by 16.38% and even surpassing the supervised TRACK-THEN-SUM approach.