Clinical AgentOps: Runtime Governance for Autonomous AI Healthcare Agents

Abstract

Generative AI is entering clinical practice not as a predictor but as an actor. Contemporary healthcare deployments increasingly involve agents: language-model systems that plan over multiple steps,retrieve patient context, invoke tools, write to the electronic health record, and coordinate with otheragents. Health AI governance, however, remains overwhelmingly design-time. Premarket review,transparency labels, and reporting standards evaluate a model artifact under the assumption that behavior is a stable property of that artifact. Agentic systems violate this assumption: their effective behavioris constituted at runtime by the composition of instructions, retrieved context, tool affordances, mem ory, and inter-agent interaction, none of which is fixed at approval time. This paper proposes ClinicalAgentOps, a framework that relocates governance from the artifact into the agent’s execution path. Itcontributes an explicit argument from the premises of design-time assurance to the necessity of inpath control, stated with its falsifying conditions; a runtime failure taxonomy for clinical agents inwhich the unit of analysis is the action trajectory rather than the input–output pair; a two-dimensionalmodel treating autonomy as a graduated, revocable, per-action grant indexed by a Clinical Action RiskTier, with a stated derivation rule from which the minimum control set for each (tier, autonomy) pairfollows; a five-plane reference architecture spanning authorization, execution, observation, assurance,and accountability; a clinical agent trace schema extending emerging generative-AI telemetry conventions with attribution, evidence, and oversight attributes, together with governance metrics computablefrom it; and a mapping from framework components to obligations under prevailing risk-management, privacy, and medical-device regimes. This is a framework and position paper: claims about controlefficacy are advanced as falsifiable hypotheses with the study designs that would test them, not asresults