Multi-Phase, Multi-Method Usability Evaluation of an Enhanced Dosimetry Quality Assurance Checklist in Radiation Oncology: From Think-Aloud Testing to Near-Live Clinical Implementation
TL;DR
This study evaluated an enhanced DQC using Borycki and Kushniruk’s multi-phase, multi-method usability evaluation framework, which integrates cognitive and socio-technical perspectives to create a “safety net” against usability problems and technology-induced errors.
Abstract
The treatment planning stage accounts for most reported patient safety events in radiation therapy, and automated quality assurance (QA) checklists are a common countermeasure; few of these tools, however, undergo formal human factors evaluation before they are released into clinical use. Building on our prior work demonstrating suboptimal usability of an institutional dosimetry QA checklist (DQC) and its participatory, theory-driven redesign, this study evaluated an enhanced DQC using Borycki and Kushniruk’s multi-phase, multi-method usability evaluation framework, which integrates cognitive and socio-technical perspectives to create a “safety net” against usability problems and technology-induced errors. Three phases were carried out in sequence at one academic medical center: (1) rapid think-aloud usability testing with two cohorts of dosimetrists and physicists (n = 10) separated by an improvement cycle; (2) remote simulation-based testing in which dosimetrists (n = 7) worked through ten high-fidelity synthetic treatment plans containing embedded errors; and (3) six weeks of near-live testing with weekly iterative refinement, involving 15 interview participants and 21 users who provided in-tool feedback (dosimetrists, physicists, and trainees). Reported usability, usefulness, and safety issues decreased by 49% between think-aloud cohorts (59 to 30), and perceived usability met recommended standards (System Usability Scale > 80). Simulation-based testing was feasible; perceived usability, completion time, embedded-error performance, and perceived realism all significantly favored easy over hard plans (participant-level exact Wilcoxon signed-rank tests, p ≤ 0.031; rank-biserial correlation 1.00). Near-live testing surfaced predominantly deeper usefulness and safety issues (77% of 142 coded comments), and the new checklist accounted for 43.6–67.5% of all checklist runs across user groups (relative use frequency). The framework provided complementary, progressively deeper insights and readied the enhanced DQC for clinical implementation.