Multimodal Trust Assessment in Human–AI Shared Control Systems

Abstract

Trust plays an important role in human–AI shared-control systems, yet it is difficult to assess from a single observable signal. This study proposes an event-centered multimodal framework for assessing situational trust by integrating contextual, behavioral, affective, and individual features surrounding critical human–AI interaction events. A controlled autonomous-driving experiment was conducted with 15 participants under high- and low-reliability conditions, yielding 117 valid event-level observations with retrospective trust ratings. The results showed substantial differences in reported trust between reliability conditions and across failure types, while facial and failure-related contextual indicators were among the more informative variables. However, the highly imbalanced trust classes, particularly the limited number of high-trust observations, constrained reliable classification of individual trust levels. The findings only provide preliminary evidence that event-level multimodal observations can characterize information associated with reported situational trust, while also demonstrating the challenges of cross-user trust estimation under limited and imbalanced data. Therefore, the study serves as a methodological and empirical starting point for developing more robust multimodal trust assessment approaches, rather than as a validation of a generalized real-time trust prediction system.