AI-Powered Smart Glove for Real-Time Hand Signto-Text and Voice Translation Using Flex Sensors and TinyML

Abstract

Sign language remains the primary mode of communication for many deaf and hard-of-hearing individuals, yet portable devices capable of real-time sign-to-text or sign-to-speech translation are still scarce and costly. This paper presents a low-cost, standalone smart glove that performs complete gesture recognition on an ESP32 microcontroller without reliance on a host computer or cloud service. Readings from five flex sensors and a six-axis inertial measurement unit (three-axis accelerometer and three-axis gyroscope, MPU6050) are combined through feature-level fusion prior to classification. Recognition is performed entirely on-device using a quantized multilayer perceptron (MLP) executed with TensorFlow Lite Micro. For comparison, a lightweight one-dimensional convolutional neural network (ID CNN) operating on raw sensor windows was also evaluated; it yielded a modest 0.8-percentage-point accuracy gain at approximately twice the inference latency, which motivated the selection of the MLP for deployment. The glove recognizes 36 static ISL/ASL alphabet and digit gestures and transmits recognition results over Bluetooth Low Energy (BLE) to a companion Android application that displays bilingual (English and Tamil) text, with optional speech output through a DFPlayer Mini audio module. A one-time, per-user calibration routine stores offset values in the ESP32's non-volatile storage (NVS), removing the need to retrain the model for each new user. With a total hardware cost below USD 18, the system achieved 91.4% cross-user recognition accuracy across five volunteer signers performing 36 gesture classes, at an average inference latency of 148 ms.