Exploring Directional Sparsity and Cross-Modal Contrast-Consensus: MERGE-Net for HSI–SAR/LiDAR Joint Classification

Abstract

Multimodal satellite imagery can provide more comprehensive information, which shows a new perspective for observing Earth’s surface. Particularly, the joint use of hyperspectral image (HSI), synthetic aperture radar (SAR), or light detection and ranging (LiDAR) has gained widespread attention due to the concurrent acquisition of spectral–spatial information, structural information, or ground elevation. However, most existing works do not explicitly learn effective intramodal and intermodal mutual information. This induces information redundancy and curtails feature diversity, stalling performance advances across heterogeneous modalities. To address the above challenge, a unified mutual learning with geometric equilibrium network (MERGE-Net) is proposed for HSI–SAR/LiDAR collaborative classification. Concretely, a modality-specific structured sparse modeling module is first designed for intramodal mutual learning, which selectively samples structurally representative neighbors using an anisotropic, axial-favoring sampling prior, thereby suppressing redundancy while enhancing discriminative spectral–spatial cues. Then, a continuous contrast-consensus integration (C3I) mechanism is put forward for intermodal mutual learning, where complementary information and consistent semantics are iteratively propagated to align heterogeneous modalities at multiple semantic depths. Finally, a geometric equilibrium equiangular tight frame head is devised to suppress minority collapse and stabilize decision boundaries in label-scarce remote sensing data. In this way, MERGE-Net can simultaneously explore intramodal and intermodal mutual relationships while enhancing discriminative and balanced class separation, offering a new pathway to multimodal classification. Experiments and analysis on three benchmark datasets implicate the effectiveness of the proposed approach compared with state-of-the-art (SOTA) methods. The code will be released at: https://github.com/jqyang22/MERGE-Net upon publication.