Domain-Adaptive Vision–Language Learning for Semantic Remote Sensing Image–Text Retrieval

Abstract

Geographic vision–language retrieval from remote sensing images demands models that can associate fine-grained visual cues with natural-language descriptions, but domain mismatch between web-scale pretraining corpora and Earth observation imagery, together with the scarcity of captioned remote sensing data, remains a significant challenge. This article studies a domain-semantic CLIP adaptation protocol for the practical setting in which a small captioned benchmark is available together with a larger label-only Earth observation corpus. The protocol combines RSITMD contrastive fine-tuning, BigEarthNet label-prompt supervision, covariance-level domain alignment, batch-level semantic prototype regularization, inference-time semantic reranking, and token-level explanation. Rather than assuming that label-only supervision is uniformly beneficial, the study isolates its effect against zero-shot CLIP and RSITMD-only fine-tuning. Experiments on RSITMD (4743 images with 23 715 descriptions) use BigEarthNet-medium as supplementary label-only supervision. On the held-out RSITMD test set of 452 images and 2260 captions, zero-shot CLIP obtains 9.51% image-to-text (I2T) R@1 and 8.76% text-to-image (T2I) R@1, while one-epoch RSITMD fine-tuning improves these values to 13.72% and 14.96%, respectively. The domain-semantic variant obtains the highest I2T R@1 (14.16%), corresponding to a 4.65 percentage-point gain over zero-shot CLIP, but it lowers T2I R@1 relative to RSITMD-only fine-tuning. These results show that label-only remote sensing supervision can strengthen scene-level semantic retrieval, but can also introduce directional negative transfer when the auxiliary labels lack caption-level spatial and compositional detail.