Development of an Indonesian Visual-to-Speech System Based on Yolov11, Qwen2.5, and MobileNetV4 for the Visually Impaired

Abstract

Visually impaired individuals face significant barriers in perceiving their surroundings and conducting cash transaction independently due to their visual limitations. This study develops an android-based assistive system that integrates three capabilities into single pipeline: object detection, contextual visual description in Indonesia, and Indonesia Rupiah Identification, with text-to-speech output, YOLOv11n and YOLOv12n were compared for object detection, Qwen2.5-VL under zero-shot prompting was compared against fine-tuned BLIP-2 for visual description, and MobileNetV4 with a CBAM attention module was fine-tuned for identifying eleven Rupiah note and coin types (seven banknotes and four coins). Evaluations on 5,000 MS-COCO validation images showed YOLOv11n achieving mAP0.5 of 0.5230 with an inference speed of 70.1 FPS, roughly 2.5 times faster than YOLOv12n despite the latter’s marginally higher accuracy, making YOLOv11n the preferred choice for a responsive assistive pipeline. For visual description, Qwen2.5-VL with YOLO detected object labels as prompting hints reached a BLEU-1 score of 0.3720 and CIDEr of 0.1621, outperforming fine-tuned BLIP-2, whose OPT-2.7B backbone and limited 32-token Q-Former query capacity constrained its ability to generate fluent Indonesian captions. The two-stage MobileNetV4+CBAM currency identification module achieved a localizer mAP0.5 of 0.9891 and a classifier overall accuracy of 0.9879 with a macro F1-score of 0.9903, with classification errors concentrated almost entirely among low-value coins rather than between coins and banknotes. These results demonstrate that the Rupiah currency detection module can run entirely on-device without requiring an internet connection, while the object detection and visual description components rely on server-side processing. This hybrid architecture balances low-latency, offline-capable currency recognition with the computational demands of more complex vision-language tasks, offering a practical assistive solution for the visually impaired. Keywords: assistive technology computer vision image captioning object detection Rupiah currency identification