Multimodal Takeover Requests for Drivers with Hearing Loss: Implications for AI-Enabled Communication in Automated Vehicles

Abstract

More than 430 million people worldwide live with disabling hearing loss. Although people with hearing loss are legally permitted to drive and may benefit from conditionally automated vehicles, SAE Level 3 systems still require drivers to respond to takeover requests when automation reaches its limits. Existing takeover requests often rely on auditory information, yet little evidence addresses visual and tactile designs for drivers who cannot rely on sound. This driving-simulator study with 40 participants examined the effects of information type (instructional, informative, and baseline), signal type (visual, tactile, and visual-tactile), and hearing condition (normal hearing and simulated hearing impairment) on takeover performance. Information type significantly affected reaction time, with baseline displays producing the shortest times. Signal type significantly affected reaction and takeover time, with visual-tactile displays producing the shortest times. The interaction between signal type and information type was significant for all three measures. Visual-tactile displays produced the shortest reaction times within every information type. With visual-tactile signaling, simple baseline alerts prompted the fastest reactions and the most abrupt maneuvers, whereas informative content produced the lowest mean maximum resulting acceleration. Hearing condition showed no significant main effect on any measure. These findings suggest that AI-enabled vehicles can support urgent takeover communication through visual-tactile displays and can adapt message content to the time available and the maneuver quality required, with implications for drivers across hearing abilities.