Advanced Frameworks for Low-latency Teleoperation and LLMdriven Autonomous Exploration in Omnidirectional Mobile Robotics
TL;DR
A robust low-latency teleoperation system that integrates extended reality (XR) with embedded robotics and the integration of a large-language-model-driven agentic exploration framework that enables natural-language-guided autonomous navigation using semantic mapping and finite-state machine orchestration are explored.
Abstract
This study presents the design and implementation of a robust low-latency teleoperation system that integrates extended reality (XR) with embedded robotics. This system bridges the gap between high-level XR development and low-level embedded control by interfacing an AjnaXR headset with a custom-built 4-wheel drive Mecanum robot. This framework is important because inherent latency and network instability in wireless human–robot interaction historically cause cognitive dissonance and severe limitations in operational safety. To address these challenges, the solution utilizes a centralized star network topology employing a multicast domain name system for reliable device discovery without relying on static Internet protocol addressing. The robot architecture was decoupled into two independent ESP32-S3 microcontroller nodes: a control node handling asynchronous motor commands and a vision node dedicated to streaming the quarter video graphics array video. The system implements a direct input-mapping paradigm, bypassing traditional ray-casting user-interface interactions, to minimize the input-to-actuation delay. The experimental results demonstrated a control latency of approximately 50 ms and a stable first-person view video stream at 20–25 frames per second. In addition, this study explores the integration of a large-language-model-driven agentic exploration framework that enables natural-language-guided autonomous navigation using semantic mapping and finite-state machine orchestration.