Real-Time ISL to Text Interpretation and Parkinson’s Tremor Correction with Augmented Reality Support

Abstract

Communication between Indian Sign Language (ISL) users and people who do not understand sign language can be difficult, whereas involuntary hand movements can make gesture-based interactions more challenging. This work presents a real-time assistive system that combines ISL alphabet recognition, landmark-based temporal stabilization, and augmented-reality (AR)-style visual feedback. A webcam provides the input stream, and MediaPipe Hands obtains 21 three-dimensional hand landmarks from each detected hand. The landmark representation is converted into 128 × 128 skeleton-based samples and classified using a convolutional neural network implemented with TensorFlow. The project dataset contains 500 images for each of the 26 ISL alphabet classes, yielding 13,000 images in total. For the reproducible quantitative evaluation reported here, a stratified 80:20 train test split with random_state = 42 was used, yielding 10,400 training and 2,600 held-out test images. The CNN achieved 99.27% accuracy, with a macro precision, macro recall, and macro F1 score equal to 99.27%. Live processing was benchmarked over 100 webcam frames at 108.40 ms per frame, corresponding to approximately 9.23 FPS. MediaPipe landmark processing accounted for 38.30 ms per frame, and CNN inference accounted for 69.73 ms per frame. The runtime implementation uses a confidence threshold of 0.85 and a temporal stability rule requiring 20 consecutive identical predictions before a letter is appended. Shaky-motion testing was qualitative, and AR-style output was demonstrated in the working prototype. The system is presented as an assistive recognition prototype and not as a medical treatment or diagnostic system.