By aligning EEG with layer-wise neural visibility rather than fixed high-level semantics, the proposed framework improves both retrieval accuracy and image reconstruction in EEG-based visual decoding.
Abstract
Decoding visual perception from electroencephalography (EEG) is important for non-invasive brain-computer interfaces (BCIs). However, most existing visual decoding pipelines directly align EEG features with semantic features from pretrained vision models. Those EEG signals carry information at more than one level and this practice disregards the varying neural visibility of different visual components in EEG signals, leading to cross modal mismatches and incomplete information use. In this work, we address this limitation through layer-wise contrastive learning. For each subject, the intermediate CLIP layer that maximizes retrieval performance is selected as the Neural Visibility Optimal Layer (NVOL). Built on NVOL, a hierarchical framework couples retrieval and generation through a shared intermediate representation. The retrieval branch fuses multi-NVOL features, aligns them to image embeddings via contrastive learning, and applies cross-domain similarity local scaling (CSLS) at test time to mitigate hubness. The generation branch reconstructs subject-specific NVOL features from EEG using a conditional diffusion prior, maps them to CLIP space through a lightweight adapter, and drives a pretrained Stable Diffusion XL model. Experimental validation on THINGS-EEG showed that, NVOL-based retrieval achieves 78.1\% mean Top-1 accuracy in 200-way retrieval, rising to 86.4\% with CSLS. Two-stage NVOL-to-semantic reconstruction also outperforms single-stage final-layer diffusion on semantic and structural metrics. By aligning EEG with layer-wise neural visibility rather than fixed high-level semantics, the proposed framework improves both retrieval accuracy and image reconstruction in EEG-based visual decoding.
The Residual Patch Adapter (RPA) is introduced, a lightweight, modular adapter that leverages all patch tokens from an intermediate layer of a ViT visual encoder for alignment, establishing design principles for leveraging the latent space of visual encoders, and open new directions for brain--image alignment and non-i...
The modeling hypothesis, supported by retrieval behavior, suggests that computational representations aligning with salient and structurally meaningful content better match macroscopic neural responses than fine-grained image details.
Yuan-Peng Li, Xie Tan, Min-Jie Tan et al.· International Journal of Neu...· 0 citations
Electroencephalography (EEG)-based Brain–Computer Interfaces (BCIs) have emerged as a promising paradigm for decoding neural activity and translating brain signals into meaningful outputs. Among the various applications of BCIs, reconstructing or generating visual stimuli from neural signals represents a challenging an...
S. S· Journal of Intelligent Decis...· 0 citations
Proposed Progressive Contrastive Alignment (ProCA), a unified and model-agnostic framework for adaptive neural-semantic alignment that progressively refines class-level contrastive supervision from frozen vision-language priors to EEG-aware semantic relations, and introduces structure-consistent interpolation to constr...
Kang-Lei Zhou, Chun-Yan Lan, Dong-Yang Li et al.· 0 citations
This work proposes a training-free representation refinement that aligns frozen embeddings without updating either encoder, which preserves the spatial information of the Perception Encoder, converts its patch grid into a compact set of learned visual views, and aggregates them for each image with a block-structured, c...
This work proposes Subject Coordinate Recovery (SCORE), a target label-free framework combining recovery-aware source training with coordinate alignment at deployment that brings brain-based visual decoding closer to robust, practical, low-latency deployment across users.
Zhenyao Cui, Siyuan Kan, Siyang Li et al.· 1 citation