Skip to content
Open access

Multiagent Large Language Model Framework for Psychotherapy Fidelity Assessment in Motivational Interviewing and Cognitive Behavioral Therapy Training: Cross-Sectional, Simulation-Based Evaluation Study

Aug 2026 · JMIR Medical Education · Vol 12, pp. e92964-e92964 · 0 citations · 60 references
Medicine

TL;DR

This proof-of-concept study suggests that a rubric-guided multiagent LLM framework can score stylized synthetic MI and CBT transcripts along expected prompt-defined competence gradients and align with preliminary external and human-rater benchmarks.

Abstract

Abstract Background Psychotherapy training is difficult to scale because manual rating of motivational interviewing (MI) and cognitive behavioral therapy (CBT) sessions is time-intensive, requires trained raters, and is subject to rater variability. Large language models (LLMs) may support simulation-based training and rubric-guided scoring, but early-stage evidence is needed before such systems can be applied to real learners. Objective This study aimed to conduct a simulation-based evaluation of a multiagent LLM framework for generating and scoring stylized MI and CBT training encounters. Methods We conducted a cross-sectional evaluation of a multiagent framework comprising Student, Patient, Evaluator, and Feedback agents. The Student agent conducted synthetic MI or CBT encounters with Patient agents derived from structured profiles. The Evaluator agent scored transcripts using study-specific MI and CBT scoring forms. Internal discrimination was tested across prompt-engineered novice, intermediate, and expert Student agent profiles using 133 MI and 102 CBT Patient profiles. Preliminary external grounding was assessed using 133 annotated motivational interviewing (AnnoMI) transcripts with coarse, metadata-derived, high/low session-level quality labels. Agreement with a pragmatic human-rater benchmark was evaluated using 16 independent human raters per modality. Criterion-level comparisons used paired Wilcoxon signed-rank tests with Benjamini-Hochberg false discovery rate correction at q<0.05. Agreement analyses used intraclass correlation coefficients (ICCs) with bootstrap 95% CIs. A secondary prompt-augmentation sensitivity analysis tested whether appending criterion-referenced feedback text to the novice Student-agent prompt shifted subsequent Evaluator-assigned scores. Results Evaluator scores increased across prompt-defined Student-agent competence levels. For MI, overall mean scores increased from 1.18 (95% CI 1.16‐1.20) for novice profiles to 1.75 (95% CI 1.69‐1.81) for intermediate profiles and to 3.57 (95% CI 3.48‐3.66) for expert profiles. For CBT, overall mean scores increased from 0.83 (95% CI 0.78‐0.88) to 2.18 (95% CI 2.10‐2.26) and 4.38 (95% CI 4.28‐4.48), respectively. Criterion-level planned contrasts were significant after false discovery rate correction. On AnnoMI transcripts, Evaluator scores aligned with metadata-derived high/low session labels, with 91.7% classification accuracy at the prespecified threshold. Human interrater reliability was an ICC(2,1) of 0.866 for MI and 0.769 for CBT. Evaluator-vs-human-consensus agreement was an ICC(2,1) of 0.959 for MI and 0.934 for CBT. In the secondary prompt-augmentation analysis, overall MI scores shifted from 1.18 to 1.44, and CBT scores shifted from 0.83 to 1.01. Conclusions This proof-of-concept study suggests that a rubric-guided multiagent LLM framework can score stylized synthetic MI and CBT transcripts along expected prompt-defined competence gradients and align with preliminary external and human-rater benchmarks. The study is innovative in separating synthetic learner, patient, scoring, and feedback-generation roles within a single simulation workflow, extending prior LLM work from plausible dialogue generation toward rubric-linked scoring. The findings support further development of scalable simulation tools for psychotherapy training research. Prospective studies with human trainees, standardized or real patients, and formal psychometric testing are required.

Read PDF

Similar papers

Review Oct 2026

Scaling Therapy Supervision: Assessing Psychotherapy Fidelity Using Large Language Models

Psychotherapy fidelity assessment measures how faithfully a therapist adheres to a treatment protocol, a critical determinant of positive patient outcomes. However, fidelity assessment is bottlenecked by manual expert review, rendering the process labor-intensive and unscalable. To scale therapy supervision and support...

Arvind Pillai, Alexis Roach, Jason Holden et al. · 0 citations
Open access Sep 2026

A Preliminary Study of Motivational Interviewing Training Using a Large Language Model (ChatGPT): Effects on Empathy and Motivational Interviewing Confidence among Social Work Students

Background: Motivational interviewing (MI) requires repeated practice, individualized feedback, and ongoing coaching that shortterm group education cannot sufficiently provide. Large language models (LLMs) such as ChatGPT may offer realistic conversational simulation and immediate feedback. This pilot study examined wh...

H. Kang, Sohyun Woo, H. Kim · 0 citations
#artificial intelligence Preprint Sep 2026

MACBT: A Multi-Agent Cognitive Behavioral Therapy Decision Support System with Longitudinal Memory

A clinician-facing AI decision-support system that combines a multi-agent CBT framework (MACBT) with a CBT-specific longitudinal memory module (CD Memory) that improves session quality and achieves a longitudinal mean of 2.29 on cross-session continuity, intervention progression, and personalization.

Deng-Du Jiang, Shuo Zhang, Wei-Wei Liao et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Evaluating Scaffolding-Oriented Multi-Agent Large Language Model System for Clinical Interview Training

Clinical education must prepare medical students to conduct safe and coherent patient interviews under conditions of uncertainty. Traditional standardized patient (SP) training is resource-intensive and difficult to scale. We developed a scaffolding-oriented multi-agent Large Language Model (LLM) AI Standardized Patien...

Lu-Ming Yang, Hao-Xian Liu, Si-Qing Li et al. · 0 citations
Open access Feb 2026

Modeling of Therapist-Client Dynamics in Psychotherapy Using LLM-Based Assessments: An Empirical Study.

BACKGROUND Psychotherapy is a primary treatment for many mental health conditions, yet the dynamic interplay among therapist behaviors, client responses, and the therapeutic relationship remains difficult to disentangle at scale. Traditional process research relies on labor-intensive human coding, limiting the ability...

Angela Chen, Si-Wei Jin, Can-Wen Wang et al. · 2 citations
Open access Sep 2026

Sustaining therapist motivational interviewing skills with a feedback system augmented by machine learning.

INTRODUCTION Following initial training, Motivational Interviewing (MI) skills tend to decline over time without ongoing feedback or coaching. However, providing ongoing support is challenging, as practitioners have had to rely on time- and cost-intensive approaches, such as human-coded fidelity feedback, to assess the...

Kristen M. West, Katherine Axford, Darin Carver et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.