Explanations from ChatGPT: Targeted Task Improvements but No Generalized Gains in Metacognitive Abilities for Intermediate Language Learners

Abstract

Metacognition is crucial for self-directed second language acquisition, yet fostering it remains challenging for intermediate learners. Explainable Artificial Intelligence (AI), such as ChatGPT, offers a novel approach by providing transparent feedback, potentially scaffolding learners' metacognitive processes. This study investigated the impact of ChatGPT's explainability on intermediate English language learners' metacognitive awareness, task accuracy, and overall writing performance. A quasi-experimental design was employed with 50 intermediate Iranian EFL learners randomly assigned to an explainable ChatGPT feedback group (n = 25) or a non-explainable feedback group (n = 25). Over a five-week intervention, participants composed argumentative essays and received automated feedback. Metacognitive awareness was measured using the Metacognitive Awareness Inventory (MAI), task accuracy was calculated based on error correction rates, and writing performance was evaluated using the TCAP/WA rubric. Data were analyzed using independent samples t-tests and repeated measures ANCOVA. Results indicated no significant differences between the two groups in generalized metacognitive awareness or overall writing performance. However, the explainable group demonstrated significantly higher accuracy on specific target tasks covered in the system's explanations. These findings suggest that while explainable ChatGPT effectively strengthens the understanding of specific, explained content, it does not confer immediate, generalized gains in metacognitive abilities or holistic writing outcomes. Future research with larger, more diverse samples and longer durations is warranted to fully ascertain the pedagogical potential of explainable AI in fostering self-regulated language learning.