Salient-Q: A Saliency-Guided Vision-Language Framework for Medical Image De-Identification

Abstract

Medical image de-identification is critical for artificial intelligence research in the healthcare domain. It requires the detection and localization of Protected Health Information (PHI) burned into medical images. While vision-language models have shown strong visual understanding capabilities, recent studies reveal they suffer from attention dispersion, which correctly localizes but fails to perceive small visual details. Moreover, naive feature selection approaches that discard spatial context destroy positional information essential for accurate bounding box prediction. We propose Salient-Q, a saliency-guided vision-language framework that addresses these challenges through three architectural innovations: (1) a Saliency Module that learns per-token PHI probability and amplifies relevant features through adaptive soft gating; (2) a position-aware scout detector module with explicit 2D positional embeddings that preserves spatial relationships, and (3) a saliency-weighted cross-attention with coordinate supervision that aligns attention centers with ground-truth bounding box centers. We also developed a digital-decayed PHI synthesis pipeline and constructed the Decayed-PHI-50K dataset that was used to fine-tune and evaluate the developed models. Salient-Q outperforms other baseline models in terms of PHI identification and localization. Our code is available at: https://github.com/zjsuper/phi\_deidentification\_vlm.