Skip to content
Book Open access

SHIFT: Step-wise History-aware Instruction Focus Tracking for Vision-and-Language Navigation

Nov 2026 · 0 citations

Abstract

Vision-and-Language Navigation (VLN) requires an agent to ground different parts of an instruction as navigation progresses. Existing VLN agents typically revisit the full instruction at each step, leaving the locally executable span to be inferred implicitly through cross-modal attention or history representations. We propose SHIFT, a lightweight instruction-side module that explicitly tracks a shared instruction focus state over tokens. This state consists of a focus center, a focus spread, and a focus summary, and is reused across two stages of step-wise grounding. Focus-Guided Context Adaptation biases language-side reading toward the currently relevant region while preserving access to the remaining instruction context, and Focus-Consistent Gating gates language hidden features according to their consistency with that same focus. By explicitly estimating and reusing one shared focus state, SHIFT coordinates instruction-side context exposure and focus-consistent language feature gating under a unified step-wise control signal. The module can be integrated into transformer-based VLN backbones without changing the task formulation. On R2R, REVERIE, and RxR-English, SHIFT yields consistent gains over strong baselines across standard navigation, goal-oriented object grounding, and long-horizon instruction following.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.