A lightweight hybrid network with inverted residuals and state space model for remote sensing image segmentation

Abstract

In current research on semantic segmentation, the majority of studies primarily focus on utilizing traditional CNNs and Transformer-based architectures. However, CNNs are limited in capturing long-range dependencies and accurately modeling global context information because their receptive fields are confined to local regions. Although Transformer can effectively achieve global context modeling through the self-attention mechanism, overcoming the limitations of local receptive fields, they inevitably introduce a computational burden that grows quadratically with the input size. In recent years, Mamba based on state-space models (SSMs) has received increasing attention due to its strong ability to capture global context while maintaining low computational complexity. However, in the field of remote sensing image segmentation, existing Mamba-based network architectures seldom achieve a balance between modeling long-range dependencies and extracting finegrained local features. Therefore, this paper proposed IRUM-Net, a novel hybrid architecture that integrates Mamba and CNN for semantic segmentation of remote sensing images. Specifically, the encoder of IRUM-Net adopted inverted residual blocks (IR Block) as its basic building units, it is designed to extract rich image features with low computational cost. Subsequently, we proposed a decoder module named Multi-Scale Local and Global Mamba Module (MLGM). This hybrid module effectively combines Mamba and CNN, allowing for the simultaneous modeling of global contextual dependencies and multi-scale local spatial details. At the same time, a Gated Enhanced Fusion (GEF) module was designed. It can connect and fuse the output of the previous decoder with the corresponding feature map in the encoder's skip connection at the same stage through a gating mechanism. This process yielded a refined feature representation enriched with multi-level information, which is then fed into the subsequent decoder stage. We conducted extensive experiments on two public datasets, Vaihingen and Potsdam, achieving 81.9% mIoU and 92.2% OA on the Vaihingen dataset, and 84.5% mIoU and 94.2% OA on the Potsdam dataset. Moreover, the overall network only has 1.98M parameters and 6.53G floating-point operations. The experiments demonstrate that the proposed IRUM-Net outperforms most semantic segmentation networks while maintaining low computational overhead, achieving excellent segmentation accuracy.