Edge AI-Based Object Detection via Voice Recognition with an LLM-Based Emotional Assistant for Elderly Care Robots

Abstract

This paper presents a fully integrated, real-time assistive system that combines voice-based object recognition with a generative conversational interface, specifically designed to enhance elderly care through edge AI deployment. The proposed framework enables intuitive human–robot interaction in domestic environments by fusing natural language understanding, optimized visual detection, and local generative response. Voice commands are processed through a speech-to-text pipeline using the Google Web Speech API, with keyword extraction triggering object detection via a quantized YOLOv8n model accelerated through TensorRT with FP16 inference on an NVIDIA Jetson Nano. In parallel, a locally deployed generative AI assistant, executed entirely on-device, provides empathetic dialogue to support social engagement and emotional well-being. The proposed system adopts a hybrid edge architecture in which object detection, robot control, and LLM-based dialogue generation are executed on-device, while speech-to-text transcription relies on a cloud-based service. This generative interface is implemented as an LLM-based Emotional Assistant. The system achieves 13 FPS with an inference latency of 70 ms for object detection, 94.3% speech recognition accuracy, and an F1-score of 0.69 at a 0.5 confidence threshold. All AI components are executed on-board, preserving privacy for on- device processing while maintaining real-time responsiveness. Experimental validation confirms the effectiveness of deploying multimodal AI, including generative models, on resource-constrained hardware. This work lays the foundation for autonomous, voice-guided care robots that not only assist in locating objects but also engage users socially, promoting greater autonomy and quality of life for older adults.