AI Errors and Human Complacency: The Output Can Be Tragic

Abstract

ABSTRACT:There’s no argument about the speed of AI – it’s fast. And sometimes, in research and healthcare, fast is wanted and needed. But fast should not compromise the integrity of the AI outputs. Research performing organizations and technology developers are promoting the use of AI tools including Large Language Models (LLMs) as an aid to the research process. Clinicians are using such tools to help inform their clinical decision-making. In research and healthcare, some errors, if not detected, could be irreversible or catastrophic. Exploring two real-life examples, alarm is raised about a potentially deadly combination: AI errors + human complacency. The role of operationalized AI Ethics Principles (safety-by-design, transparency, human-in-the loop, and accountability) is emphasized as a form of risk mitigation.