Adaptive Sparse Cross-Scale Transformer for Computationally Efficient Brain Tumor MRI Segmentation

Abstract

Brain tumor segmentation from Magnetic Resonance Imaging (MRI) is a critical task in medical image analysis, owing to the irregular morphology of tumors, heterogeneous intensity distributions across MRI modalities, and the presence of multiple overlapping sub-regions. Accurate delineation of these regions is essential for clinical diagnosis, treatment planning, and longitudinal disease monitoring. Although Vision Transformer (ViT)-based architectures have demonstrated promising performance in medical image segmentation by capturing long-range dependencies and global contextual information, conventional multi-scale transformer models remain constrained by static grouped attention mechanisms that introduce redundant computations and exhibit quadratic complexity with respect to input size. To address these limitations, this paper proposes the Adaptive Sparse Cross-Scale Transformer (ASCT), a computationally efficient framework for brain tumor MRI segmentation. ASCT incorporates three key innovations: (i) a Dynamic Scale Routing (DSR) module that adaptively weights multi-scale features using learned routing coefficients, replacing fixed channel grouping; (ii) a Sparse Token Attention (STA) mechanism that restricts attention computation to the most informative token pairs, reducing complexity from quadratic O(N²) to near-linear O(Nk); and (iii) a linearized attention approximation that significantly reduces GPU memory consumption during training. Additionally, cross-scale feature fusion is performed prior to the attention operation to suppress redundant computations and enhance inter-scale feature interaction. The proposed ASCT model is evaluated on the BraTS 2021 dataset for multi-region segmentation, targeting Whole Tumor (WT), Tumor Core (TC), and Enhancing Tumor (ET) sub-regions. Experimental results demonstrate that ASCT achieves Dice scores of 89.2%, 84.1%, and 83.5% for WT, TC, and ET, respectively, yielding an average Dice score of 85.6%. Compared to the baseline Vision Transformer, the proposed model reduces computational complexity from 94.6 GFLOPs to 58.4 GFLOPs and GPU memory usage from 11.2 GB to 7.3 GB, confirming its efficiency and practical viability for real-world clinical applications.