Skip to content
Preprint

Prototypes of the Mind: A Unified Framework for Probing the Visual Brain

Nov 2026 · 0 citations · 44 references
Biology Computer Science

TL;DR

This work introduces a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives, and establishes prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain.

Abstract

Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective---such as neural encoding, stimulus decoding, or post hoc interpretation---making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives. Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout. This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel's selectivity. Using this interface, we identify which visual concepts drive neural responses and where they appear in the image, reveal cortical organization through voxel tuning in prototype space, and perform controlled stimulus synthesis to probe cortical selectivity through targeted interventions. Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain.

View source

Similar papers

Sep 2026

The Visual Target Matters: Learning across the Visual Hierarchy for Brain-to-Image Retrieval

NeuroGlyph, which learns a trial-independent visual target from multiple depths of a frozen visual backbone, achieves the strongest system-level performance in six of eight reported metrics, and support learning retrieval targets across the visual hierarchy rather than prescribing one visual depth.

Ye Wang, Hao-Kun Ren, Hong Yu et al. · 1 citation
#artificial intelligence Preprint Oct 2026

StimFlow: Synthesizing Brain Activity from Images via Stimulus-conditioned Flow Matching

The inherent variability of fMRI responses across trials encourages a generative view of the stimulus-to-response mapping. Prior generative encoders often operate in compressed latent spaces and condition predominantly on semantic features, which provide limited low-level cues for early visual cortex. We propose StimFl...

Hung Van Bui, D. Vu, Hoang-Son Vo et al. · 0 citations
Open access Nov 2026

The representation of object-concept primitives across the macaque brain.

The results provide a blueprint for how the brain encodes object concepts and suggest how concept representations engage perceptual representations through feedback during vision.

Spencer R. Loggia, Helen E. Feibes, S. Duffield et al. · 0 citations
Preprint Nov 2026

Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks

Understanding how neurons respond to visual stimulus is an important question in both deep learning and neuroscience. It has significant implications for enhancing the interpretability of black-box artificial neural networks and for understanding the visual representation in the human brain. We propose a structured vis...

Yu-Song Zhao, Edgar Y. Walker, Hao Wang et al. · 0 citations

Cross-attention encoding models reveal dynamic spatiotemporal routing across human higher visual cortex

This work uses per-parcel cross-attention to dynamically route features from a self-supervised video model (V-JEPA-2) across both space and time, and finds that joint routing improves predictions of brain responses to held-out videos across higher visual regions, most consistently in lateral and dorsal visual areas ass...

Iishaan Inabathini, Margaret M. Henderson · 0 citations
Preprint Nov 2026

AVID: A 5T fMRI Dataset for Benchmarking Auditory-induced Visual Mental Imagery Decoding

Visual mental imagery (VMI) decoding uses brain activity to reconstruct internally generated visual scenes, providing a potential pathway for externalizing internal experiences and developing neuroprosthetic communication interfaces. However, progress is limited by the lack of dedicated, training-scale VMI datasets. Cu...

Shi-Qi Shen, Shu-Rui Li, Xi-Lin Zhang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.