Exploring the efficacy of AI-powered conversational agents (APCA) in developing English speaking proficiency with a focus on fluency and intelligibility amongst ESL learners
Abstract
This study explores the efficacy of AI-powered conversational agents (APCA) in developing English speaking proficiency with a focus on fluency and intelligibility among ESL learners. The study adopted a mixed-methods exploratory design involving 32 beginner to intermediate ESL learners randomly assigned into Control and Experimental groups. Over an intervention period of 8–12 weeks, participants engaged in structured speaking practice sessions. The Control group received conventional instructor-led speaking practice, while the Experimental group used APCA tools, namely ChatGPT and Microsoft Copilot, as substitutes for human tutoring. Learners’ speaking performance was assessed through pre-test, post-test, and followup test phases based on fluency and intelligibility criteria. Inter-rater reliability analysis using Quadratic Weighted Cohen’s Kappa demonstrated substantial to almost perfect agreement between raters across most assessment phases. Mann–Whitney U tests revealed no statistically significant differences between the Control and Experimental groups in fluency and intelligibility scores across all testing phases, suggesting that APCA-supported speaking practice may be comparably effective to conventional human-guided practice. In addition, qualitative findings derived from thematic analysis indicated that learners perceived AI feedback positively, particularly in enhancing speaking confidence, motivation, and willingness to practice independently in a low-anxiety environment. However, participants also highlighted limitations such as occasional inaccuracies, robotic responses, and lack of emotional support. Overall, the findings suggest that APCA can function as an effective supplementary tool for improving speaking proficiency and learner confidence in ESL contexts, although it should complement rather than replace human interaction in language learning.