HumanSpeech-TeleOp: Human-in-the-Loop Edge-IoT Framework for Speech and LLM-Driven Robot Teleoperation and Object Detection

Abstract

This paper proposes HumanSpeech-TeleOp, a novel human-in-the-loop Edge-IoT framework that enables humans use speech to interact with the robotic system to perform remote robotic teleoperation and object detection tasks. The human user uses a raspberry pi based user interface to record and send speech commands to the remote backend processing node for speech recognition and keyword extraction to convert it into a semantic command format. The semantic command is sent to the robotic edge node, which identifies the command type and performs robot teleoperation or object detection tasks based on the type of command given by the user. Initial experimentation conducted in the lab environment showed that the system recognized and performed robot teleoperation and object detection tasks, given through user speech command, with 100% accuracy and 100% success rate with an average latency of around 2 to 3 seconds.