Global and Neighbor-Aware Token Learning for Weakly Supervised Remote Sensing Image Semantic Segmentation

Abstract

Image-level weakly supervised remote sensing semantic segmentation aims to learn pixel-level land-cover prediction using only image-level labels, greatly reducing the annotation cost of fully supervised methods. Class activation map (CAM)-based methods are widely used for this task, but they usually focus on the most discriminative regions, leading to incomplete activation and inaccurate boundaries. Recently, vision Transformer (ViT)-based methods have been introduced to alleviate the limitation of CAMs by exploiting token relations and attention mechanisms. However, remote sensing images often contain dense land-cover regions with subtle interclass differences, and patch tokens in deep ViT layers may become oversmoothed without explicit patch-level supervision, weakening local semantic discrimination. Moreover, large intraclass variations and frequent category co-occurrence make image-specific class tokens prone to semantic drift across different remote sensing images. To address these problems, we propose a global and neighbor-aware token learning (GNATL) framework. GNATL contains two complementary modules: neighbor-aware patch token learning (NPTL) and global class token learning (GCTL). NPTL exploits overlapping regions between neighboring crops to construct implicit patch-level constraints, thereby alleviating patch token oversmoothing. Global class token learning (GCTL) dynamically maintains global class tokens as category-level prototypes to guide image-specific class tokens toward stable category semantics. Experiments on the International Society for Photogrammetry and Remote Sensing (ISPRS) Potsdam, ISPRS Vaihingen, and DeepGlobe Land Cover datasets show that GNATL achieves mean Intersection over Union (mIoU) scores of 56.38%, 47.74%, and 62.75%, outperforming the best compared methods by 2.63%, 4.57%, and 2.36%, respectively.