Epistemically-Aware AI: Toward a Bayesian Framework for Human-AI Complementarity

Abstract

Human–AI teams in healthcare can, in principle, achieve complementarity, performing better than either constituent alone. Yet empirical evidence consistently shows this potential is rarely realized (Jacobs, Pradier, McCoy, Perlis, Doshi-Velez, and Gajos, 2021; Vaccaro, Almaatouq, and Malone, 2024). I argue that this complementarity gap cannot be fully understood or resolved within the dominant, performance-based conception of complementarity, which defines success in terms of decision accuracy and thereby fails to capture the underlying epistemic process through which human–AI collaboration succeeds or fails. I propose a Bayesian framework that reconceptualizes complementarity at the level of credences: a human–AI system is epistemically complementary when the credences formed on the basis of both human and AI signals are more accurate than those formed on the basis of either signal alone, as measured by proper scoring rules. Within this framework, I distinguish Potential Epistemic Complementarity (PEC), the maximal epistemic gain achievable under ideal Bayesian integration, from Realized Epistemic Complementarity (REC), the gain actually achieved in practice. I further introduce a decomposition of the realized posterior into three distortion parameters capturing well-documented empirical phenomena, and use this framework to derive design principles for AI-driven decision support systems (AI-DSS) that maximize epistemic complementarity for the specific clinician and context at hand. A crucial implication of this perspective is that AI-DSS should be epistemically-aware, modelling the knowledge and context of the clinician they are teamed with.