HCTNet: a hierarchical hybrid convolution-transformer network with bidirectional cross-attention fusion for brain tumor MRI classification

Abstract

Accurate brain tumor classification from magnetic resonance imaging (MRI) requires modelling both fine-grained local cues, such as tumor margins and intensity heterogeneity, and long-range contextual cues, such as the spatial relationship between the lesion and the surrounding anatomy. Convolutional neural networks (CNNs) capture the former but are limited by restricted receptive fields, whereas vision transformers (ViTs) capture the latter but underuse fine local structure. Existing hybrids typically combine the two by sequential stacking or late concatenation at the network tail, so local and global representations are not progressively aligned and seldom exchange information bidirectionally. We propose HCTNet, a four-stage hierarchical hybrid network in which a convolutional Local Feature Extractor (inverted-residual blocks with squeeze-and-excitation) runs in parallel with a transformer-based Global Feature Extractor (spatial-reduction self-attention at high resolution, standard multi-head self-attention at low resolution). The two branches are coupled by the proposed Cross-Attention Fusion Module, invoked after every paired block, which performs bidirectional cross-attention followed by a learnable channel-wise gated combination, so the representations are aligned and re-weighted throughout the backbone rather than only at the classifier head. For interpretability, we compute Grad-CAM and Integrated Gradients jointly on a shared explanation set and report per-image and per-class agreement (mask intersection-over-union and heatmap Pearson correlation). On the full canonical four-class Brain Tumor MRI benchmark (5, 712 training and 1, 311 held-out test slices), HCTNet attains 97.79%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$97.79\%$$\end{document} overall accuracy, a macro-averaged F1 of 0.977 and a macro-averaged one-vs-rest AUC of 0.998 with only 16.0M parameters. Under an identical from-scratch training protocol it matches or exceeds representative CNN, transformer and recent (2022-era) backbones—including ResNet-50, EfficientNetV2-S, ConvNeXt-T, Swin-T and Swin-V2-T—while using fewer parameters, and an ablation confirms that the proposed bidirectional Cross-Attention Fusion Module outperforms additive and concatenation fusion. The interpretability analysis further shows that Grad-CAM and Integrated Gradients expose complementary region-level and pixel-level evidence concentrated within the brain parenchyma.