Bidirectional Entity Coupling Alignment for Fine-Grained Remote Sensing Image–Text Retrieval
Abstract
Due to the homogeneous backgrounds and cluttered layouts of remote sensing (RS) imagery, fine-grained local entities are indispensable for preventing cross-modal semantic mismatch. However, existing parameter-efficient fine-tuning (PEFT) methods based on contrastive language-image pretraining (CLIP) encapsulate scenes into holistic global representations, diluting these critical local cues. To address this, we propose the bidirectional entity-level alignment network (BEAN), a unified fine-grained entity alignment paradigm. First, to avoid the semantic ambiguity of conventional extraction mechanisms, we introduce a symmetric visual and textual entity construction paradigm comprising a visual entity bank constructor (VEBC) and a noun-to-token index constructor (NTIC). Specifically, VEBC constructs a visual entity bank via layerwise extraction, while NTIC performs index-guided extraction of textual entity embeddings, jointly delivering semantically discriminative cross-modal entity pairs. Second, to counteract the submersion of local entities within global embeddings, we design the cross-modal entity-bidirectional adapter (CEBA). It drives bidirectional cross-attention between the global feature stream and localized entity tokens. This mutually enriches isolated entities with holistic scene context, while simultaneously reinjecting fine-grained entity cues back into the global representations, effectively reversing the dilution process. Third, to overcome the absence of explicit entity-level supervision in traditional global contrastive objectives, we propose the fine-grained entity contrastive loss, a sample-level formulation that provides direct supervision for explicit entity alignment without requiring manual positive–negative pair annotation. Experiments on RSITMD, RSICD, and UCM-Captions datasets demonstrate that BEAN achieves leading mR scores of 50.74%, 34.67%, and 60.56%, respectively, consistently outperforming both PEFT-based methods and the fully fine-tuned CLIP baseline across all three benchmarks. The code is available at https://github.com/JasonGravity/BEAN