A Disease-Memory Guided Framework for Radiology Report Generation from Multi-View Chest X-rays

Abstract

Chest X-rays are widely used, low-cost diagnostic tools. Writing chest X-ray reports manually is a time-consuming and tedious task. The increasing number of X-rays burdens radiologists, creating a need for a clinically accurate, fast and automated report generation system. To solve this challenge, we propose a new disease-memory-guided vision-language framework for radiology report generation.  Frontal and lateral chest X-rays (multi-view) are encoded using a Swin-Transformer. These encoded visual features are then passed through a multi-label classification head that predicts disease probabilities, from which the top three labels are converted into learnable disease-memory embeddings.  These are concatenated with projected visual tokens and fixed task instruction embeddings to form a multimodal prompt which is fed into a frozen decoder. A LLaMA-2-7B model adapted to the radiology domain using LoRA fine-tuning on the train-only MIMIC-CXR subset is used as an autoregressive decoder. The disease-memory embeddings serve as disease guidance during report generation. An extensive experimental analysis of different experimental variants is conducted on the benchmark Indiana University Chest X-ray dataset using three aspects: 1) statistical n-gram overlap, 2) semantic similarity, and 3) clinical label-based metrics. Experimental results of the proposed system show improvements in semantic evaluation metrics, with BERTScore precision, recall, and F1 values of 0.898, 0.891 and 0.894, respectively, and ROUGE-L and METEOR scores of 0.327 and 0.281, indicating improved contextual consistency between generated and reference reports. However, performance on statistical metrics (e.g., BLEU4=0.068) and clinical evaluation metrics remains limited, with Micro-F1, Macro-F1, Precision and Recall values of 0.4211, 0.0640, 0.4272, and 0.4151, respectively, indicating only partial abnormality description. This framework improves abnormality grounding while maintaining computational efficiency and can be used as a supportive tool for preliminary report drafting in research or resource-constrained settings.