Lightweight real-time image captioning for mobile systems: an optimized multimodal framework and benchmarking study

Abstract

Visually impaired people continue to face barriers in accessing written and audio content. To address this, we propose a comparative study of advanced technological solutions that enhance autonomy and environmental interaction. Specifically, we utilize deep learning models, including InceptionV3, InceptionV4, and ResNet50, in conjunction with long short-term memory (LSTM) and gated recurrent unit (GRU) decoders, all of which are trained on the MS COCO 2017 dataset. Furthermore, integrated OCR, translation, and speech synthesis modules improve visual recognition and multilingual accessibility. These capabilities are brought together in a mobile application that combines these features for real-time interaction. For evaluation, we use task-specific captioning metrics, a composite operational score, and mobile-oriented efficiency indicators. The optimized InceptionV4–LSTM model achieves a BLEU-4 score of 0.4189, a ROUGE-L score of 0.688, and a SPICE score of 0.0599 for image captioning. After converting the trained model into a frozen TensorFlow ProtoBuf (.pb) format and deploying it within the Android application on a Samsung Galaxy A11 smartphone, the model achieves an inference time of 178 ms, a frame rate of 5.6 FPS, and a memory usage of 410 MB. The full multimodal pipeline, evaluated on 3,451 images, reaches an overall operational score of 92.6%, showing favorable performance compared with the evaluated baseline configurations. Taken together, these findings support the feasibility of efficient multimodal assistive systems for real-world mobile accessibility.