Deep Learning-Based Indian Sign Language Alphabet Recognition Using Convolutional Neural Networks

Abstract

Indian Sign Language (ISL) identification is an important aspect in increasing the communication of deaf or hard-of-hearing (DHH) people with the hearing community. Transfer learning is applied with a deep learning system based on EfficientNet-B0 for the automatic recognition of ISL letters. We used a publicly available ISL Alphabet Dataset, which includes 26 static letter classes (A to Z), for both training and evaluation. The image size was set to 224 x 224 before the preprocessing that involves normalization, data cleaning, and data augmentation like rotation, zoom, translation, altering brightness, and changing contrast to increase robustness and generalizability. In this, the dataset is separated into training, validation, and testing sets using a 70:15:15 split. By replacing the classification head with fully connected layers and adding a Softmax output layer for multi-class classification, EfficientNet pretrained on ImageNet is fine-tuned. It is a fact proven by experimental results that when tested against the proposed model, the test data had a precision of 98.34%, recall of 98.18%, and accuracy of 98.21%, as well as an F1-score of 98.25%. From analysis with a confusion matrix, one can see that the model distinguishes almost all ISL alphabet gestures except for similar-looking hand signs, which cause minor confusion. Further, analysis shows that the proposed approach is better off than traditional CNN VGG16, MobileNetV2, and ResNet50, while the computational costs remain low.