Multi-Granularity Text and Scale-Aware Visual Encoding for Remote Sensing Image-Text Retrieval and Prompt-Based Scene Recognition

Abstract

Remote sensing captions typically interleave domain-specific terminology, object attributes, spatial relations, and scene context within a single sentence, while the corresponding imagery contains targets spanning markedly different spatial extents—from individual vehicles to entire airport layouts. A single global text vector and fixed-scale visual pooling therefore tend to discard the local and hierarchical evidence needed for reliable image–text alignment. Moreover, standard contrastive objectives treat all unmatched pairs as equally unrelated, ignoring the fact that semantically adjacent scenes (e.g., airports and industrial zones) share substantial visual content. To address these issues, we develop a CLIP-style dual encoder with three targeted additions: a text encoder that preserves word-level, phrase-level, and sentence-level representations before fusing them through cross-granularity attention; a visual encoder that associates four hierarchical feature levels with scale-aware receptive-field configurations and incorporates SAM-guided local views during training; and a semantic membership-aware contrastive loss that applies detached, bounded discount weights to semantically close negatives without inverting their penalty sign. On the cleaned RSICD benchmark, the model achieves 87.5% mAP for in-domain prompt-based scene recognition and 34.37% mean recall for scene-constrained retrieval, outperforming the quoted RemoteCLIP and fully fine-tuned CLIP results in the scene-constrained retrieval comparison. Ablation confirms that each of the three components contributes meaningfully to the final performance. The deployable dual encoder contains approximately 59.7 million trainable parameters, and SAM is not required at inference.