Beyond the Black Box: Unraveling the Role of Explainability in Human-Artificial Intelligence Collaboration

Abstract

Explainable artificial intelligence (AI) models have been proposed to mitigate overreliance and underreliance on AI, which reduce the effectiveness of human-AI collaborative tools. Yet, empirical evidence is mixed, and the impact of explainable AI on the cognitive effort and fatigue of a decision maker (DM) is often overlooked. This paper offers a theoretical perspective on these issues. We develop an analytical model that incorporates the defining features of human and machine intelligence, capturing the limited but flexible nature of human cognition with imperfect machine recommendations. Crucially, we represent how AI-based explanations influence the DM’s belief in the algorithm’s predictive quality. Our results indicate that explainable AI has varying effects depending on the level of explainability provided. Although low explainability levels have no impact on decision accuracy and reliance behavior, they lessen the cognitive burden on the DM. In contrast, higher explainability levels enhance accuracy by improving overreliance but at the expense of increased underreliance. Further, the relative impact of explainability is higher when the DM is more cognitively constrained, when the decision task is sufficiently complex, or when the stakes are lower. Importantly, higher explainability levels can escalate the DM’s cognitive burden (and hence, overall processing time and fatigue) precisely when explanations are most needed (i.e., when the DM is pressed for time to complete a complex task and doubts the machine’s quality). Our study clarifies how explainability affects decision outcomes and cognitive effort, informing the design of effective human-AI systems across decision environments. This paper was accepted by Jeannette Song, operations management. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2024.07450 .