An Integrated Deep Learning Framework for Intelligent Navigation Assistance to Visually Impaired Individuals Using CNN-Based Object Detection, RNN Based Scene Understanding, and Image Segmentation

Abstract

Navigation is an important problem for people with visual impairment. This problem has been affecting the independence and safety of the visually impaired in unfamiliar areas. This paper introduces a new deep learning-based smartphone-based navigation assistant for the visually impaired. The proposed system is named “Aurora”. The system has been designed specifically for assisting the visually impaired with navigation and obstacle avoidance. The system uses convolutional neural networks and recurrent neural networks for accurate interpretation of the visual information obtained by the smartphone’s camera. The proposed system also uses semantic segmentation for the detection of sidewalks, pedestrian crossings, obstacles, and signages. The proposed system also uses audio and haptic feedback for safe and efficient navigation. In this paper, the proposed system’s architecture and the dataset preparation are discussed. The training and optimization of the proposed system are also discussed. The proposed system uses the smartphone’s GPS and inertial sensors for improved accuracy. The proposed system has been tested in various environmental conditions. This proposed system has proved to be very accurate for detecting obstacles and giving instructions on how to navigate through them. One more aspect of Aurora’s user-oriented approach is that it is possible to change the mode of feedback depending on the needs of users. This paper will also continue discussing the usability assessment with visually impaired users, and it has revealed improvement in both the confidence of navigation and reduction of travel time. The findings indicate that Aurora can greatly assist visually impaired people