Skip to content
Open access

Stress-testing consentAI: a pipeline for evaluating a conversational educational adjunct to surgical informed consent

Oct 2026 · npj Digital Surgery · Vol 1 · 0 citations · 29 references

Abstract

Informed consent is delivered inconsistently, and the comprehension it should produce is often poor. We built ConsentAI, a conversational educational adjunct that guides patients through a surgeon-authored consent form and gates progression on demonstrated understanding. Before any patient-facing trial, we asked how such an agent can be evaluated at scale, and which methods can be trusted. Eight clinicians scored 50 simulated laparoscopic cholecystectomy sessions on a 10-item rubric. We tested four LLM judge architectures against this clinician reference under a pre-specified, imbalance-robust plan. No single judge reproduced clinicians’ failure calls. Only jury aggregation improved sensitivity (0.421; 95% CI, 0.26–0.61), and no architecture reached a usable triage point. A constant pass-everything labeler topped raw agreement and AC1 yet scored zero on Matthews correlation and Cohen’s kappa. Evaluating clinical conversational agents therefore requires multiple clinician ratings and imbalance-robust metrics, treating clinician adjudication as a measurement layer, not ground truth.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.