SMaRTAban: a voice-controlled LLM agent for quadruped mobile robotics with integrated vision

Abstract

Large Language Models (LLMs) have emerged as powerful tools for enabling intuitive human-robot interaction through natural language. However, existing approaches often lack evaluated voice control in languages other than English, integrated vision, and comprehensive safety mechanisms. We present SMaRTAban (Smart Multi-legged Robotic Transformer Agent Artaban), a system that combines a LangGraph ReAct agent with the quadruped mobile robot Artaban for natural language control. Our system integrates: (1) voice commands with automatic language detection via Whisper, evaluated on English and Slovak, (2) on-demand scene interpretation via GPT-4o vision, (3) a safety-focused system prompt with interruptible tool execution, and (4) a React-based GUI for transparent operation monitoring. A pilot voice-control evaluation on 120 samples showed robust semantic understanding across English and Slovak (BERTScore F1: 0.87–1.00), despite transcription challenges in Slovak. We further conducted 4,800 accepted tool-calling trials across two models, six agent and prompting configurations, two languages, and 10 command scenarios. The strongest configuration combined iterative ReAct execution, an explicit system prompt, and few-shot prompting with three task-specific movement examples, achieving a tool-call F1 of 0.9977 and 96.5% exact-sequence success with GPT-4o. A separate performance evaluation showed that remote transcription and model processing dominated response time, whereas local ROS communication contributed only a few milliseconds.