Counterfactual Trust-Aware Explainability for Multimodal AI Decision Systems

Abstract

With the growing use of multimodal AI systems in healthcare applications, conversational AI, affective computing, and decision support systems, there is an increased demand for explainable and trusted AI algorithms. But current methods used to explain AI systems typically result in attribution-based explanations without any causal reasoning abilities, perturbation resistance, and consistent explanations across different modalities. This paper presents a new method called Counterfactual Trust-Aware Explainability (CTAE) that aims at developing an explainability framework for multimodal AI decision systems. Our approach integrates multimodal counterfactual reasoning, cross-modal consistency, perturbation resistance, trust calibration, and human-oriented evaluation into a unified explainability framework. Evaluation of the framework was conducted on the MELD, IEMOCAP, and CMU-MOSEI benchmark datasets based on measures like explanation fidelity, perturbation stability, contradiction rate, semantic consistency, causal alignment, and human trustworthiness perception. The experimental results showed that the CTAE framework yielded greater scores in explanation fidelity (0.892), perturbation stability (0.874), and trustworthiness (4.61) when compared to methods like SHAP, LIME, attention visualization, and traditional counterfactuals. Additionally, CTAE demonstrated lower contradictions and better performance in flipping decisions by constraint-based perturbations. Finally, human-centered evaluation of the explanations generated by the CTAE framework confirmed improved quality, usability, and trustworthiness of the explanations across multimodal interaction settings. Overall, the proposed CTAE framework provides a robust and trustworthy explainability solution for high-stakes multimodal AI applications requiring transparent and cognitively reliable decision interpretation.