Navigating AI Errors: The Role of Mental Models in Human-AI Team Performance
TL;DR
It is demonstrated that error pattern effects are more evident in users’ interpretations of system behavior than in their detection performance, which advances theoretical models of monitoring and mental model development in AI-supported decision environments.
Abstract
Performance of human-AI teams depends on both AI’s overall accuracy and human’s ability to detect errors and form an accurate understanding of system behaviors. Little prior research has compared how different patterns of AI errors (i.e., when and how they occur) shapes users’ mental models that underpin human oversight. To address this gap, we conducted an online experiment ( N = 380) exploring how error patterns influence individuals’ ability to detect AI errors and develop effective mental models of system accuracy in a phishing email classification task. Our findings demonstrate that while different error patterns did not significantly influence participants’ ability to detect AI errors, it shaped their perceptions of how the AI performed over the interaction. This work demonstrates that error pattern effects are more evident in users’ interpretations of system behavior than in their detection performance. This distinction advances theoretical models of monitoring and mental model development in AI-supported decision environments.