This work introduces a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives, and establishes prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain.
Abstract
Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective---such as neural encoding, stimulus decoding, or post hoc interpretation---making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives. Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout. This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel's selectivity. Using this interface, we identify which visual concepts drive neural responses and where they appear in the image, reveal cortical organization through voxel tuning in prototype space, and perform controlled stimulus synthesis to probe cortical selectivity through targeted interventions. Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain.
NeuroGlyph, which learns a trial-independent visual target from multiple depths of a frozen visual backbone, achieves the strongest system-level performance in six of eight reported metrics, and support learning retrieval targets across the visual hierarchy rather than prescribing one visual depth.
Ye Wang, Hao-Kun Ren, Hong Yu et al.· arXiv.org· 1 citation
The inherent variability of fMRI responses across trials encourages a generative view of the stimulus-to-response mapping. Prior generative encoders often operate in compressed latent spaces and condition predominantly on semantic features, which provide limited low-level cues for early visual cortex. We propose StimFl...
Hung Van Bui, D. Vu, Hoang-Son Vo et al.· 0 citations
The results provide a blueprint for how the brain encodes object concepts and suggest how concept representations engage perceptual representations through feedback during vision.
Spencer R. Loggia, Helen E. Feibes, S. Duffield et al.· Current Biology· 0 citations
Understanding how neurons respond to visual stimulus is an important question in both deep learning and neuroscience. It has significant implications for enhancing the interpretability of black-box artificial neural networks and for understanding the visual representation in the human brain. We propose a structured vis...
Yu-Song Zhao, Edgar Y. Walker, Hao Wang et al.· 0 citations
This work uses per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, and finds that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas ass...
Iishaan Inabathini, Margaret M. Henderson· arXiv.org· 0 citations
Visual mental imagery (VMI) decoding uses brain activity to reconstruct internally generated visual scenes, providing a potential pathway for externalizing internal experiences and developing neuroprosthetic communication interfaces. However, progress is limited by the lack of dedicated, training-scale VMI datasets. Cu...
Shi-Qi Shen, Shu-Rui Li, Xi-Lin Zhang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.