Effective Implementation of XAI in Healthcare: Results from a Survey with Non-Technical Users

Abstract

The growing integration of AI in healthcare has increased demands for transparency, trust, and accountability. XAI tools address these needs by providing interpretable insights into AI-generated outcomes. However, there is still a lack of empirically grounded guidance on how to implement XAI tools effectively in healthcare organizations and how to integrate them properly into existing clinical processes and inter-departmental workflows. This study examines the main challenges in implementing XAI tools in healthcare and pinpoints to recommendations for addressing them. Using a mixed-methods approach of stakeholder survey and expert interviews, we analyzed XAI use, implementation, integration, adoption barriers, and possible solutions. Our findings reveal six key challenges, including the misalignment between explainability outputs and clinical reasoning, inadequate technical readiness, limited organizational support in terms of permissions and financial resources, and the need for structured training and legal compliance. Our analysis of the survey data also yielded six practical recommendations as candidates for good practices for implementation and organizational integration. Finally, we conducted an initial perceptionbased evaluation of these recommendations with experts.