A Transformer-Based Deep Learning Framework for Real Time Sign Language Recognition and Multilingual Translation

Abstract

Communication has been an important aspect in all the human lives. Especially for persons with disability to communicate linguistically, sign language comes to the rescue. Bridging the gap between the signers and non-signers is essential. To enable the signers to express in a language becomes essential while communicating to those who are non-signers. In this paper, a deep learning based complete pipeline that recognizes the ISL gestures, transforms the gloss words to complete meaningful English sentences is presented. Furthermore, the system includes a module for translation from English to Tamil, a regional Indian language. A dataset is created by the team similar to Indian Sign Language. Dataset to augment the original. The hand features are extracted using the MediaPipe framework, and saved as separate CSV files for words and alphabets. A Multi-Layer Perceptron model is built to classify these words and alphabets. The gloss words recognition has an accuracy of 98.65% and the alphabet recognition has an accuracy of 99.96%. After the real-time sign language video is converted as sequence of words, a pre-trained T5 transformer is used to build meaningful and grammatically correct sentences. Using a pre-trained model for English to Tamil translation the English sentences are translated to Tamil.