Multi-Task Visual Perception Network with LLM Conditioning for Autonomous Navigation
TL;DR
This work proposes a user-friendly, interactive framework that eliminates the reliance on globally consistent maps and integrates visual perception with Large Language Models (LLM) to interpret user commands via text or voice.
Abstract
Long-term navigation for service robots faces critical challenges like the accumulation of odometry drift and sensor error, which progressively degrade 2D maps and renders traditional path planning algorithms (e.g., A*, RRT*, DiPPer, ViT-A*) ineffective over time. To address this, we propose a user-friendly, interactive framework that eliminates the reliance on globally consistent maps. Our approach integrates visual perception with Large Language Models (LLM) to interpret user commands via text or voice. Instead of relying on a drift-prone global map, the system generates a sequential action plan based on local visual cues and egocentric geometric instructions. These action plans are executed sequentially, allowing the robot to navigate known and unknown environments safely. By resetting localization relative to immediate targets, our framework effectively works with a minimum accumulation drift strategy, ensuring accurate, efficient, and collision-free navigation without the maintenance overhead of traditional mapping. Experiments on real-world and simulated data have shown significant improvements over other methods. Our source code is publicly accessible at https://github.com/PraveenSingh24/VL-Navigation.