UG-FPR: uncertainty-guided feature perturbation and refinement for medical foundation models

Abstract

Abstract. Vision Transformers pretrained via self-supervised learning have demonstrated strong representation capability in natural image analysis and are increasingly adopted in medical imaging tasks. However, when transferred to medical domains, pretrained encoders often exhibit limited adaptability to ambiguous anatomical boundaries and low-contrast structures as contextual dependencies learned from natural images may not adequately capture uncertainty characteristics inherent in medical data. In this work, we propose a layer-wise uncertainty-guided feature perturbation and refinement framework that operates directly in representation space and can be seamlessly integrated into pretrained transformer encoders. The proposed method explicitly estimates spatial uncertainty from encoder features and performs controlled semantic diffusion in feature space, enabling selective refinement of ambiguous regions while maintaining stable representations elsewhere. The enhanced representations are integrated through residual modulation, enabling progressive adaptation without disrupting pretrained dynamics. The proposed framework is fully plug-and-play and can be inserted into each transformer layer without modifying the original architecture. Extensive experiments on multiple medical image analysis tasks demonstrate consistent performance improvements over strong transformer baselines, validating the effectiveness of uncertainty-guided representation diffusion for medical visual understanding.