HuGDAT: Guiding Transformer Attention with Human Vision

Abstract

Vision Transformers (ViTs) achieve state-of-the-art performance across various computer vision tasks but often lack the interpretability and biological plausibility of human vision. Unlike the human cognitive system, which efficiently filters complex environments to focus on salient regions, ViT attention mechanisms frequently disperse across irrelevant input areas. While prior research has used human eye fixation data for saliency prediction, its utility as a direct guidance signal for transformer-based architectures remains underexplored. We present the Human Guided Deformable Attention Transformer (HuGDAT), a study investigating the use of human eye fixation data as a guidance signal for bridging human and machine attention. Rather than treating fixation as a downstream supervision target, we integrate fixation points directly into the deformable attention mechanism. This allows the model to guide the spatial sampling of keys and values toward perceptually meaningful regions without requiring heavy architectural modifications. Across the CAT2000 and Figrim datasets, we observe consistent improvements in accuracy and more human-aligned attention patterns, especially for object-centric and visually salient categories, demonstrating the efficacy of incorporating human priors as a lightweight supervision signal promoting robust and interpretable vision transformers.