Dual-Key MoCo With Early-Stage Adversarial Fusion for PAN–MS Joint Classification

Abstract

Joint classification of multispectral (MS) and panchromatic (PAN) imagery has achieved remarkable success, aiming to provide more detailed and accurate interpretations of ground objects. However, when labeled samples are insufficient, the generalization performance of deep learning-based methods can be significantly affected. Therefore, we explore a novel cross-modal contrastive learning paradigm, the Dual-Key MoCo, to achieve few-shot joint representation learning using unlabeled samples. We also utilize an early-stage adversarial fusion approach to mitigate the loss of unique features from a single source. Specifically, 1) we propose an early-stage adversarial fusion mechanism that leverages correlation to construct adversarial fusion constraints (AFCs), thereby highlighting unique frequency information and reducing similar redundant information; 2) we propose a contrastive learning paradigm, Dual-Key MoCo, that enhances intraclass invariance and interclass discriminability by maximizing similarity between anchors and positives while minimizing similarity with negatives; and 3) we construct a cross-modal self-supervised pretraining framework that leverages sufficient unlabeled samples to develop a robust joint representation prior. This provides a robust pretraining model for few-shot PAN and MS joint classification tasks. The theoretical analyses and experimental results on multiple popular datasets comprehensively illustrate the robustness and effectiveness of our proposed method under a few-shot learning. Our source code is accessible at: https://github.com/Xidian-AIGroup190726/DKMoCo