Remote Sensing Image Captioning via Dual-Stream Fusion and Spatial Relation-Aware Encoding

Abstract

Abstract. Remote sensing image captioning (RSIC) aims to describe key objects in remote sensing images using natural language, with significant applications in disaster assessment, land-use identification, and scene understanding. Existing methods face two critical challenges: insufficient cross-modal alignment due to the domain gap between generic visual representations and remote sensing semantics, and inadequate spatial relation modeling among regions in complex scenes, which compromises the semantic precision and logical coherence of generated descriptions. To address these issues, this paper proposes the Dual-Stream Relation-Aware Transformer (DSRAT) for remote sensing image captioning. On the visual encoding side, multi-scale CNN features serve as the foundation, fused with domain-specific semantic priors from RemoteCLIP through a gated dual-stream fusion module to achieve adaptive alignment of multi-source visual information. Subsequently, a spatial relation-aware mechanism is introduced into the encoder self-attention, which explicitly encodes geometric relationships such as relative position, distance, and orientation between regions as attention biases, enhancing the model’s capability for structured representation of complex spatial layouts and multi-object interaction scenarios. Finally, adaptive weighted aggregation of multi-layer encoder outputs generates discriminative cross-modal memory representations for the decoder. Experiments on the RSICD and NWPU-Captions datasets demonstrate that DSRAT achieves state-of-the-art performance across six metrics on RSICD and all seven metrics on NWPU-Captions. In particular, DSRAT achieves a significant performance improvement of +14.45 CIDEr on NWPU-Captions compared to the state-of-the-art method, validating the effectiveness of the proposed approach.