LIP Reading Using Neural Network and Deep Learning

Abstract

Lip reading, the task of inferring spoken words purely from visual observation of lip movements, remains a challenging problem even for trained human lip readers, who are typically able to correctly identify only about every second word. This paper presents an automated lip reading system built using Convolutional Neural Networks (CNNs) to classify spoken words from video sequences of a speaker's mouth region. The system uses a Haar Cascade classifier to localize the face and mouth region in each video frame, followed by a dlib-based facial landmark detector to extract a precise lip Region of Interest (ROI). The cropped and normalized lip images are then passed to a trained CNN, which was benchmarked against a Long Short-Term Memory (LSTM) network and a Temporal Convolutional Network (TCN) to model the sequential nature of lip movement, with the CNN architecture selected for deployment based on superior classification accuracy on the Lip Reading in the Wild (LRW) dataset. The trained model was integrated into a real-time application capable of capturing live webcam video, isolating the speaker's lip region frame-by-frame, and predicting the spoken word with an associated confidence score. Experimental evaluation on a ten-class subset of the LRW dataset shows that the proposed CNN model is able to reliably identify isolated words such as “Hello”, “Start”, “Stop”, and “Previous” in real time. The system demonstrates the practical feasibility of vision-only speech recognition and its potential applications in assistive hearing technology, security and surveillance, and human–computer interaction in audio-degraded environments