This paper presents the first systematic benchmark of 17 edge-deployable SLMs against 4 online APIs for robotic navigation instruction decomposition, and proposes a lightweight hybrid semantic-geometric goal localization framework that combines open-vocabulary object detection, prompted segmentation, and LiDAR geometry to estimate metric goals.
Abstract
Foundation-model-based vision-language navigation (VLN) has advanced autonomous robot navigation by enabling robots to interpret natural-language instructions, identify semantic goals, and follow user-specified behavioral rules. However, existing VLN systems rely heavily on cloud-hosted foundation models for language understanding and semantic grounding, limiting their applicability where network connectivity is unavailable and reliable metric goal localization is required. Although recent small language models (SLMs) enable fully onboard inference, their suitability for navigation instruction decomposition has not been systematically evaluated. This paper makes three contributions toward fully onboard VLN for outdoor environments. First, we present the first systematic benchmark of 17 edge-deployable SLMs against 4 online APIs for robotic navigation instruction decomposition, evaluating accuracy and latency on human-annotated instructions across three computing platforms and providing practical guidance for selecting onboard language models. Second, we propose a lightweight hybrid semantic-geometric goal localization framework that combines open-vocabulary object detection, prompted segmentation, and LiDAR geometry to estimate metric goals, while maintaining visual bearing guidance when reliable geometric observations are unavailable. Third, we integrate these advances into Edge-BehAV, a fully onboard extension of the BehAV architecture that enables cloud-independent behavior-guided navigation. Experimental results show that the best offline SLM matches the instruction decomposition performance of the strongest cloud API while running approximately 9x faster and without network connectivity. The proposed goal localization framework reduces mean goal-distance error from 2.05 m to 0.20 m at lower computational cost, and the complete system succeeds in 31 of 32 closed-loop outdoor trials.
Vision-and-Language Navigation (VLN) tasks require an agent to interpret natural language instructions and visual observations to navigate complex environments. Existing methods mostly construct topological or semantic maps and rely on the Large Language Model (LLM) for navigation decision-making, however, they still suffer from limited adaptability to dynamic environments and robustness to complex instructions, with task performance being limited by LLM performance. To overcome these limitations, we introduce the Real-time Occupancy-aware Visual Language Map (RO-VLMap). This framework quantifies the instantaneous risks posed by moving entities in the environment by fusing real-time occupancy sensing with visual language 3D reconstruction. Specifically, RO-VLMap, when combined with our proposed Navigation Adaptive Module (NAM), leverages the complementary advantages of the Knowledge Graph (KG) and LLM to parse open-vocabulary instructions into precise navigation goals. By planning over a unified occupancy-aware map, the agent proactively generates safe paths that avoid dynamic obstacles. Experimental results show that RO-VLMap significantly improves navigation success rates and efficiency. Furthermore, it demonstrates strong robustness in unseen scenarios, providing a practical solution for autonomous navigation of embodied agents in complex real-world environments.
Yuan Liu, Chuang Hu, Nan Ding· International Conferences on...· 0 citations
Vision-Language Navigation (VLN) in unseen indoor environments is useful in real-world robotics, where an agent must follow natural-language instructions, locate objects, and answer spatial questions without a pre-built map or fixed object vocabulary. Multimodal vision-language models (VLMs) provide strong open-vocabulary grounding and zero-shot reasoning, but struggle to emit reliable metric quantities such as range, bearing, and comparative spatial relations directly from images. Existing approaches address this by folding geometry into hand-engineered pipelines or asking models to output waypoints, requiring changes to the control stack for different robots, tasks, or vocabularies. We introduce AnchorVLN, an open-vocabulary VLN system built on a simple rule: the VLM proposes semantics; geometry decides metrics. It is realised as EMBODIED-NAV-MCP, a Model Context Protocol (MCP) server driven by a VLM agent through a compact set of callable tools. Since no tool accepts distance in metres or bearing in radians, the schema enforces the semantic-geometry boundary without modifying the downstream autonomy stack. We benchmark both tasks of the CMU Vision-Language Navigation Challenge 2026: 30 instruction-following questions over 15 scenes and a frozen 45-question object-reference set. The full system achieves 64.4 percent on instruction following, dropping by 13.3 percentage points without controller modeling (t = 2.77). On object reference, geometric anchoring clears the challenge overlap threshold on 10 of 45 questions, versus 0 of 45 for direct coordinate estimation, reducing median center error from 3.37 m to 2.48 m.
Long Giang Vu, Cheng-Kai Yao, Yu-Xin Liu et al.· 0 citations
Continuous-environment vision-and-language navigation (VLN-CE) requires interpreting natural-language instructions in unseen 3D environments and executing continuous low-level actions. Existing methods often depend on LiDAR, panoramic cameras, or extra sensors; separate geometric-mapping and semantic-navigation visual representations can cause long-trajectory spatial-semantic inconsistencies. We propose LG-VLN, a monocular zero-shot framework with shared visual features and LangGraph-based state orchestration. An online feed-forward 3D reconstruction network predicts depth, camera poses, and dense point clouds for agent-pose estimation and global map fusion. Geometry and navigation share dense CleanDIFT features: semantic consistency rejects incorrect inter-frame correspondences, while target-instance constraints define visual references whose similarity combines with local BLIP-2 image-text relevance to form a semantic value map. LangGraph represents instruction parsing, geometric perception, semantic value updates, path planning, action execution, and failure recovery as a directed state graph with conditional transitions, persistent state, and modular recovery mechanisms. On a fixed 550-episode subset of the R2R-CE val-unseen split, LG-VLN achieves 21.3% success and 12.1% success weighted by path length. Ablations show shared semantic features improve navigation, further boosted by combining visual similarity and image-text relevance. Results establish shared visual representations and explicit state orchestration as effective for zero-shot VLN-CE using monocular RGB alone. Code will be publicly released for reproducibility.
Jian-He Zhao, Yan-Hua Qiu, Zhi-Yu Zhang et al.· 0 citations
Autonomous semantic indoor navigation requires robust low-level control and high-level understanding of objects and spatial context in cluttered and partially occluded environments. While deep reinforcement learning (DRL) methods such as twin delayed deep deterministic policy gradient (TD3) enable reactive obstacle avoidance, they typically struggle with long-horizon semantic navigation, where object-location memory and language-level reasoning are required. We present a lightweight hierarchical two-stage framework that, to the best of our knowledge, is introduced for the first time to integrate a locally deployed vision–language model (VLM), semantic object coordinate memory, and a TD3-based DRL controller for language-conditioned indoor navigation. In Stage 1, the robot performs semantic exploration using odometry, 2D LiDAR, and VLM-based object recognition to build a geometric map and store detected object categories with their estimated world coordinates in a structured javaScript object notation (JSON) semantic memory. In Stage 2, a natural language query is used to retrieve the target object coordinates from memory and pass them to a TD3 target point navigation policy, which performs mapless navigation using odometry and RealSense RGB-D perception. The proposed framework combines open-vocabulary VLM-based object coordinate estimation, LiDAR mapping, RGB-D perception, and language grounding within a unified semantic memory representation. Experiments in a ROS-integrated realistic simulation demonstrate consistent goal-reaching performance and improved navigation efficiency compared with an Artificial Potential Field baseline using the same VLM and a DRL + GPT-4o mini configuration. We also compare the proposed VLM-based recognition module with YOLO-World v2.6 and Grounding DINO, showing that the VLM-based approach provides more reliable semantic grounding and target-coordinate estimation in the tested indoor navigation scenarios, particularly for flexible natural language object queries.
Yernar Akhmetbek, Ayaulym Parmash, Temirlan Meiramkhanov et al.· Italian National Conference...· 0 citations
This work instantiates VoLN for aerial navigation through VoLN-UAV, a 7,210-episode benchmark that combines long-horizon goal-directed flight, continuous 3D motion, large viewpoint changes, and context-dependent beacon selection and reveals substantial remaining challenges in long-horizon evidence integration, cross-view goal matching, and closed-loop stability.
Jiabin Lou, Hao-Peng Wang, Yuan-Shuai Wang et al.· arXiv.org· 0 citations
Vision-language models (VLMs) show strong visual understanding for aerial navigation, but their action generation remains unreliable. We study this gap in the context of orbit-search-based navigation, where an aerial agent circles a reference landmark while a VLM detects a language-described goal. We find that the VLM recognizes goals with stable accuracy across configurations, yet the subsequent approach phase—where the VLM must navigate toward the recognized goal—succeeds only 11.3% of the time. The failure is not a detection problem but a navigation problem: the visual approach prompt uses a format absent from the training data, and no amount of prompt engineering restores performance. We propose Geometric Waypoint Navigation, which eliminates VLM-based navigation after detection. The aerial agent moves toward the detected direction using the same deterministic waypoint-following controller already proven in the orbit phase, requiring no additional training or parameters. On the CityNav test unseen split, success rate increases from 35.5% to 49.0%, substantially surpassing all published baselines including GeoNav (25.9%), HTNav (21.7%), and FlightGPT (21.2%). Detection rates remain exactly constant at 31.0%, confirming that gains originate entirely from improved navigation rather than improved perception. Controlled experiments show that geometric waypoint following outperforms VLM Replay by 8.0 percentage points, and a fixed 30m approach distance outperforms VLM-estimated adaptive distances by 3.0 points. Our findings suggest a broader design principle: when a learned model serves as a detector in an embodied pipeline, separating perceptual decisions from motor execution can substantially improve reliability.
Hao-Tian Xu, Chen-Xu Wang, De-Jun Chen et al.· 2026 12th International Conf...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
