Attention-Based Fusion of EEG Spectrograms and Scalograms for Schizophrenia Detection Using Vision Transformers

Abstract

Abstract Schizophrenia is a neuropsychiatric disorder that affects brain activity, making timely and reliable diagnosis challenging. Although electroencephalography (EEG) provides a non-invasive and cost-effective diagnostic modality, its non-stationary nature and complex temporal–spectral characteristics limit the effectiveness of conventional machine learning and deep learning approaches, which often rely on a single time–frequency representation or simple feature fusion strategies. To address these limitations, this study proposes an attention-based fusion framework that integrates complementary time–frequency representations for EEG-based schizophrenia detection. Spectrograms and scalograms generated using the Short-Time Fourier Transform (STFT) and Continuous Wavelet Transform (CWT) capture global spectral information and localized temporal–frequency characteristics, respectively. These representations are processed by a Vision Transformer (ViT), while a bidirectional cross-attention module enables effective interaction and fusion of complementary features before global feature learning. The proposed framework achieved average classification accuracies of 98.34 ± 0.21% and 98.68 ± 0.15% on Dataset 1 and Dataset 2, respectively. Comparative experiments demonstrate that the proposed framework consistently improves classification performance over methods based on individual time–frequency representations and conventional feature fusion approaches, indicating that the proposed attention-guided fusion strategy effectively exploits complementary EEG information for schizophrenia detection.