Towards bridging the gap between human and machine vision with representational alignments

Abstract

Compared to human vision, state-of-the-art deep neural networks (DNNs) remain more susceptible to both distribution shifts and adversarial perturbations. The robustness of human vision is believed to stem from the brain’s hierarchical and recurrent architecture, which integrates bottom-up with top-down visual processing. Inspired by this, researchers have introduced bio-inspired inductive biases into DNNs to make artificial intelligence more brain-like. These biases take two primary forms: 1) Architecture-based mechanisms, such as recurrent feedback connections and 2) Optimization- based methods, including regularization using neural data. Such approaches have been shown to enhance model robustness to various perturbations. Nevertheless, it remains unclear whether these improvements in robustness actually stem from emulating human visual processing—that is, whether the models develop brain-like internal representations that lead to human-like perceptual behavior. This dissertation directly addresses this question through three interconnected projects that probe the relationship between brain-like representations and robust, human-like perception in models. In Project I, we co-trained object recognition models to simultaneously classify images and predict human EEG responses. We demonstrate that this neural regularization significantly boosts adversarial robustness, and first identify that this improvement is specifically correlated with the model’s capacity to predict EEG signals from mid-level channels at early time points(100ms). In Projects II and III, we systematically ablate the recurrent dynamics in a predictive coding model (PVGG16) [16] to evaluate their role in emulating human neural representations. We demonstrate that these recurrent dynamics enhance the model’s alignment with neural data across scales and time: they improve its fit to human fMRI signals in higher-level visual cortex during natural scene viewing (Project II) and translate into a higher alignment with the hierarchical information present in human EEG responses over time (Project III). Notably, the differential EEG alignment for ambiguous versus intact images emerges specifically during the middle time points of the visual response, a signature of intermediate hierarchical processing. This finding underscores that recurrent dynamics are critical for capturing the brain’s divergent computations when processing ambiguous versus intact visual inputs. Collectively, this work provides insights into why bio-inspired inductive biases work: they enforce representational alignment with specific, functionally critical components of the human visual processing hierarchy. By forging a stronger bridge between neural computation and artificial intelligence, this thesis offers a framework for building more robust and interpretable machine vision systems, progressing toward the goal of approaching human-like visual processing.