Skip to content

Gaze Prompts: Temporally Dense Human Attention for Vision-Language-Action Fine-Tuning

Sep 2026 · arXiv.org · Vol abs/2609.34550 · 0 citations · 35 references
Computer Science

TL;DR

eye-tracker-supervised gaze prompting is introduced, which uses gaze recorded during VR teleoperation to provide frame-level visual guidance for VLA fine-tuning.

Abstract

Vision-Language-Action (VLA) fine-tuning pairs images with actions at every step, yet typically provides only a task-level language instruction, leaving moment-to-moment visual relevance implicit. We introduce \emph{eye-tracker-supervised gaze prompting}, which uses gaze recorded during VR teleoperation to provide frame-level visual guidance for VLA fine-tuning. During training, recorded gaze locations are rendered as crosshairs on the robot's head-camera images. At deployment, a lightweight predictor estimates gaze locations from recent images and the instruction, supplying the same type of visual prompt without an eye tracker or changes to the policy architecture. Instantiated with $\pi_0$, gaze prompting increases mean success from $26.3\%$ to $56.0\%$ across six real-world bimanual manipulation tasks, with gains also observed when a single policy is trained on all six tasks. We release \textsc{GazeMani}, a dataset of $1{,}200$ teleoperated trajectories with synchronized gaze.

View source

Similar papers

Preprint Oct 2026

GazeAct: A Human-Gaze-Annotated Manipulation Dataset for Attention-Regularized Policies

Vision-language-action (VLA) policies must ground language instructions in the right objects, contact points, and target locations for reliable manipulation. Human gaze provides a natural signal for this grounding, but existing robot datasets rarely pair real human gaze with the multi-camera observations used by modern...

Anupam Pani, Wang-Gui Yuan, Qi-Fan Wang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Eye, Robot: Learning Expressive Gaze Dynamics via a VLM-Guided Transformer-VAE

Gaze coordinates attention in human interaction through coupled eye, head, and eyelid motion. A humanoid robot must choose where to look while generating a suitable movement toward that target; fixed tracking rules alone do not model the temporal structure of human gaze. We separate these tasks with an asynchronous vis...

Yu-Hang Hu, Junlin Lyu, Yuan Fang et al. · 0 citations
Preprint Oct 2026

GazeAgent: VLM-Guided Implicit Gaze Calibration from Egocentric Multimodal Behavior for AI Glasses

AI glasses typically require explicit, user-specific eye-tracker calibration, and accuracy can drift when the device shifts during use. We present GazeAgent, a VLM-guided framework that continuously and adaptively improves gaze-calibration accuracy from naturally occurring behavior without asking users to interrupt the...

Yu-Hong Zhang, Khanh An Nguyen, Xin-Yu Tian et al. · 0 citations
Sep 2026

ActGaze: Learning Action-Grounded Gaze through Counterfactual Visual Interventions for High-Precision Manipulation

ActGaze is a training approach that guides VLA policies to gaze on task-relevant regions, much like humans gaze on critical visual cues while executing precise movements, and consistently outperforms the base VLA policy and other visual-grounding approaches.

Jin-Xuan Zhu, Jia-Heng Wang, Chao Tang et al. · 0 citations
Sep 2026

OpenVAM: Open-World Visual Attention Modeling with VLMs

Predicting human gaze is a core capability for applications ranging from web/UI design analysis to robotics and human-computer interaction. Yet, most visual attention modeling methods output only a dense saliency map, which is often insufficient for action: practitioners need to connect attention peaks to discrete elem...

Kiana Hooshanfar, A. Kazerouni, Alireza Hosseini et al. · 1 citation

Gaze-Driven Efficient Inference for Vision-Language Models

It is suggested that human gaze provides an effective and practical guidance signal for improving the computational efficiency of Vision-Language Models without requiring changes to the underlying models, offering a simple and general approach for reducing visual inference cost while largely preserving answer quality.

Unknown authors · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.