Environment-Aware Trajectory and Object-Location-Guided Refinement for Fine-grained Remote Sensing Change Captioning
Abstract
Remote Sensing Image Change Captioning (RSICC) aims to describe the differences between two images with a short description. Existing methods conventionally concentrate on optical imagery under clear-sky conditions, failing to account for complex multimodal environmental influences (e.g., clouds, haze, low-light conditions, as well as SAR and multispectral modalities). Moreover, these methods merely generate coarse-grained change captions, leading to the omission/misjudgment of critical change information, particularly in complex scenarios. To address these limitations, we propose a Fine-grained RSICC (FRSICC) task tailored for complex multimodal environments. To effectively capture critical change information under such environments, we propose an environment-aware trajectory and object-location guided refinement framework for the FRSICC task, termed FRSICC-TR. Specifically, to quantify image environmental information, we propose Environment-Aware Trajectory Generation, which generates the collaboration trajectory among different agents. We further propose Dynamic Image-Enhanced Caption Generation that dynamically adapts to image-specific environmental perturbations for targeted enhancement to generate a high-quality initial change caption. To better capture fine-grained changes, we design Object-Location-Guided Caption Refinement to iteratively recover missed changed objects and correct imprecise spatial location information. We finally construct a large-scale FRSICC dataset, called FRSI-CC. Extensive experiments demonstrate that FRSICC-TR achieves SOTA performance in fine-grained captioning.