Preventing escalation in customer service: immediate motivation and 4-week outcomes in a workplace deployment of LLM role-play training

Abstract

Many customer support employees encounter aggressive customer attitudes, including overt dissatisfaction, unreasonable demands, and verbal abuse. Such attitudes do not always stem solely from customers: inadequate employee responses can amplify dissatisfaction and escalate initially manageable issues, making skills that prevent escalation critical. Recent advances in large language models (LLMs) enable role-play training with virtual interlocutors. However, role-play systems in which virtual customer attitudes shift in response to the trainee's utterances under pedagogically grounded rules remain underexplored, and evaluations of such systems rely on reaction-level measures taken immediately after use, whose relation to later workplace outcomes has not been examined in a deployment. We developed an LLM-powered training system that provides spoken dialogues with a virtual customer whose emotional expressions adapt to the employee's utterances, paired with a virtual coach for feedback. The core of the system is a state-based dialogue framework in which the virtual interlocutor's attitude escalates or de-escalates according to five binary action flags assigned to each trainee utterance, under rules co-designed with experienced call-center practitioners. We conducted a 4-week field study in an operating customer support organization, randomly assigning twelve employees to a customer-service scenario group ( n = 6) or a structurally matched non-customer-service scenario group ( n = 6), each completing two intervention sessions 1 week apart. The largest estimate was for motivation for behavioral change immediately after use, favoring the customer service group ( g = 1.11, 95% CI [–0.33, 2.55]). Over 4 weeks, the between-group contrasts on change scores were small and slightly favored the comparison group (responsiveness g = –0.26, 95% CI [-1.55, 1.03]; productivity g = –0.24, 95% CI [–1.53, 1.05]); the corresponding within-group estimates in the customer service group were medium-to-large (responsiveness g = 0.72, 95% CI [–0.48, 1.92]; productivity g = 0.77, 95% CI [–0.45, 1.99]). All intervals include zero. The largest estimate fell on the measure whose content matched the customer-service material, and participants' comments followed the same pattern. We therefore argue that reaction-level measures aligned with training content cannot on their own evidence training effectiveness, and we report the framework and the deployment procedure as a basis for better-powered evaluations.