Realizing the value of the divergence event in clinical human-AI interaction

Abstract

When a clinical AI system’s output differs from a human clinician’s judgment, what follows is consequential and underspecified. Current AI interfaces offer only a narrow lane for what happens next: they present the AI output (and its associated potential divergence) as an adjudicative choice for a human to accept, override, or modify. In practice, collapsing this human-AI interaction into a single output inadvertently obscures what happens in a potential divergence event (Fig. 1 ) which can be characterized along three axes: What the workflow and interface allow the clinician to see and do ( structural axis), what information each side brought to the interaction ( informational axis), and how the clinician engaged with the AI’s reasoning rather than only its output ( behavioral axis). When clinical judgment and AI output diverge, clinicians must ultimately decide what to recommend and how to explain their recommendation to the patient, consider their autonomy, accountability, and liability, and bear moral responsibility 1 . A recent systematic review of AI ethics in nursing and midwifery found that AI challenges the role of health professionals not through replacement, but through subtle reconfiguration of the activities that count as care and that emphasis on AI efficiency may risk certain relational aspects of care less visible and valued 2 . Because clinicians often cannot see how an AI system reached its output, they are left to choose between that AI output and their own judgment. Clinicians are told to be the “human in the loop” but are typically given no method for doing so beyond “exercising their judgment” 3 . Efforts to make AI output both more technically explainable and human interpretable - have not directly correlated with improved clinical decisions and have sometimes created false confidence in incorrect AI output 4 .