Self-evaluative reliability: a missing safety property in adaptive clinical AI governance

Abstract

Adaptive clinical AI governance is shifting from static pre-market approval processes toward continuous post-market oversight, creating circumstances in which AI-generated self-evaluation may contribute to post-deployment safety assurance. Existing regulatory approaches, such as Predetermined Change Control Plan and Good Machine Learning Practice principles, address monitoring, modification, and performance drift but do not explicitly define the reliability of AI-generated self-evaluation as a distinct validation objective. Emerging studies of AI-based clinical evaluation show substantial variability in agreement across models, evaluators, tasks, and evaluation criteria. Although this evidence stems largely from AI-as-evaluator rather than from direct self-evaluation, it demonstrates why evaluative reliability should not be inferred from general model performance. We define self-evaluative reliability (SER) as the consistency and dependability with which an AI system assesses the quality, uncertainty, or limitations of its own outputs under predefined evaluation conditions. We propose SER as a distinct safety property that should be independently validated when self-evaluative judgments contribute to monitoring, escalation, adaptation, or continued deployment. Rather than introducing a novel regulatory framework, SER extends established principles of reproducibility and reliability to AI-generated evaluations. Establishing this validation before self-evaluation becomes load-bearing could strengthen the safety and transparency of adaptive clinical AI governance.