SignSpeak: An AI-Based Utility for Visually Impaired Individuals

Abstract

Visually impaired people (VIP) need assistive technology for smooth and confident navigation on pavements along the roads. VIPs find it hard to follow the speech generated through a text-to-audio converter due to the presence of advertisement text. Automating the process of audio conversion from the signboards by integrating the capabilities of emerging AI models in a fraction of a second will be useful for their confident navigation. In this context, SignSpeak, an AI-based assistive utility, is designed to convert signboard text to speech using visual inputs, which may be captured through smart sensors or canes. The proposed system follows a three-step (detection-extraction-conversion) pipeline. Initially, a fine-tuned lightweight YOLOv11s model is used to generate a bounded box for the relevant text region, which is input to Gemini model, a multimodal large language model for text extraction. Lastly, the extracted text is converted into audio using the Google Text-to-Speech tool. We fine-tuned the YOLOv11s model on 200 signboard images and tested its performance on an additional curated dataset of 42 real-world signboard images. Experimental findings indicate that the proposed integrated pipeline achieved superior performance compared to the solitary usage of the Gemini model for text extraction followed by its audio conversion.