PIPA: Prior-Driven Prompting With Diagnosis-Oriented Retrieval-Augmentation for 3-D Radiology Report Generation

Abstract

Automatic radiology report generation has gained increasing attention for its potential to assist in clinical reporting and reduce the workload of radiologists. Existing 3D radiology report generation methods employ multi-modal foundation model to encode volume-text inputs and produce diagnosis reports, while they ignore the characteristics of 3D volumes including much background regions and suffer from generating hallucinations, especially in medical domain that contains many uncommon professional terms. In this paper, we aim to efficiently adapt the pre-trained foundation model to specific 3D radiology report generation, and present a Prior-drIven Prompting with diagnosis-oriented retrieval-Augmentation (PIPA) framework. In PIPA, we design a Prior-drIven Prompting (PIP) strategy to exploit diagnostic knowledge from input volumes and a Diagnosis-oriented volume-report retrieval-augmentation Generation (DIG) module to explore beneficial knowledge from external database. Specifically, in PIP, to take full advantage of the patient’s clinical information, e.g., age and symptoms, and the possible disease information, e.g., brain tumor, edema, we formulate them as the patient and disease priors to mine clinical relevant knowledge. Furthermore, we propose utilizing visual and textual embeddings as queries to retrieve similar external data by devising a diagnosis-oriented retrieval-augmentation scheme for leveraging more report resources as references for LLM to produce accuracy outcomes. With PIP and DIG, PIPA integrates clinical priors and external data to learn effective diagnostic representations for high-quality report generation. We evaluate the framework on both public and in-house 3D medical datasets with corresponding reports, demonstrating its strong performance in generating accurate diagnosis reports. Source codes have been published at https://github.com/CUHK-AIM-Group/PIPA/tree/main