From Wizard-of-Oz to Autonomous: A Hybrid Rule and LLM Backchannel Architecture for a Silent Listening Agent

Abstract

We present a working autonomous system that replaces the human wizard in a previously validated listening-agent architecture for second-language (L2) production support. The system streams learner speech through cloud ASR and runs two parallel rule-based policies, one for content-driven facial affect and one for rhythm-driven head motion, alongside a third LLM-mediated meta-policy that supplies a slow coherence signal over a 15-second window. Facial expressions are grounded in FACS Action Units and rendered on a 158-morph ARKit-mapped avatar. From two real pilot sessions we identify concrete failure modes of a unified, priority-ordered classifier (1 nod in 44 decisions on a monologue) and show that the parallel policy split rebalances it (19 nods of 34 decisions on the follow-up session). We frame these two-session results as a preliminary technical evaluation of system behaviour: they characterise what the architecture does, not yet how it is perceived, and a controlled human-centred study is left as the immediate next step. Full per-session event logs and a demo video are released as an artifact for the GENEA Reproducibility Award.