Multi-modal interaction enhanced segment anything model (MIE-SAM) for RGB-T salient object detection.

Abstract

RGB-T Salient Object Detection (RGB-T SOD) effectively leverages the complementary information of RGB images and thermal infrared images to locate important targets in complex environments, such as low light, rainy and foggy weather, or cluttered backgrounds. However, the existing deep-learning based models have two key problems: the fixed fusion strategy can not adapt to the changing environmental conditions, and the scarcity of pixel-level annotation leads to over-fitting. While the Segment Anything Model (SAM) exhibits excellent generalization, its adaptation to RGB-T SOD is challenged by the lack of saliency semantics, RGB-only pre-training and manual prompt dependency. To address these challenges, we propose a Multi-modal Interaction Enhanced Segment Anything Model (MIE-SAM). The framework reconfigures SAM's image encoder into a weight-sharing dual-branch image encoders. A Multi-modal Low-Rank Adaptation module (Mm-LoRA) is embedded in the frozen dual-branch image encoders to inject saliency-specific semantics and facilitate deep cross-modal feature interaction while preserving SAM's pre-trained knowledge via parameter-efficient fine-tuning. Furthermore, a Dynamic Fusion Module (DFM) learns dynamic fusion weights to adaptively aggregate multi-modal embeddings based on their environmental reliability, ensuring robust integration in varying environments. Finally, a Progressive Decoder Module (PDM) directly translates the fused features into the fine-grained saliency map in an entirely prompt-free, end-to-end manner. Extensive experiments on several publicly available RGB-T SOD datasets show that our method achieves state-of-the-art performance, exhibiting robustness and generalization ability in a variety of challenging scenarios. The code is available at https://github.com/Aazzz66/MIESAM.