Explainable schizophrenia detection using fusion of convolutional neural network and vision transformer from electroencephalogram

Abstract

Schizophrenia (SZ) is associated with subtle alterations in neural dynamics that are difficult to capture using conventional electroencephalogram (EEG) features. This study introduces a unified deep learning framework that integrates recurrence plots (RP) with wavelet synchrosqueezed transform (WSST) representations into a single fused image modality and leverages attention-enhanced hybrid convolutional–transformer architectures for subject-level classification. Specifically, we propose RP+WSST image fusion combined with convolutional neural network (CNN)–vision transformer (ViT) hybrids (ResNet-18–ViT and EfficientNet-B0–ViT) to jointly model local spatial patterns and global contextual dependencies. Subject-wise 10-fold cross-validation and a strictly isolated hold-out protocol (70/15/15 split) are employed to prevent subject leakage and provide unbiased performance estimates. Compared with single-backbone CNN and ViT models, the proposed hybrid architecture demonstrates competitive generalization. Interpretability is enhanced using appropriate XAI. Gradient-weighted class activation mapping (Grad-CAM) for the CNN branch and Attention Rollout for the ViT branch. The proposed framework is applied to automated SZ detection from resting-state scalp EEG using two independent databases (Warsaw and Atieh schizophrenia EEG (ASEEG)). On the Warsaw dataset, the ResNet-18–ViT hybrid achieved 82.14% accuracy (AUC 83.58%) using Amor-based WSST features. On the ASEEG dataset, performance reached 95.04% accuracy with AUC values above 95%. Channel-wise analysis identified frontal, temporal, and central electrodes as the most discriminative regions, consistent with known SZ-related electrophysiological abnormalities. These findings demonstrate the practical feasibility of deploying the proposed explainable AI (XAI) framework for reliable EEG-based clinical decision support in psychiatric engineering applications.