Effectiveness of ChatGPT Feedback on Low-Level English Students’ Writing Proficiency and Their Perceptions: An Experimental Study

Abstract

The purpose of this study is to explore the effectiveness of ChatGPT’s feedback on the writing of non-English major university students and their perceptions of this implementation. Participants in this study had a low level of English (A2). Researchers employed an experimental research design to achieve the research objectives. There were 42 students in total, including 21 students in the experimental group (EG) and 21 in the control group (CG). EG received ChatGPT feedback after writing draft 1, whereas CG received peer feedback and general instructions from teachers.  Data were collected through pre- and post-tests, and semi-structured interviews. Paired sample t-tests revealed a significant improvement in mean scores between pre-and post-tests of EG (p=.00<.05), in contrast to CG’s. While students complimented ChatGPT on its immediacy, dialogic, and personalized responses, the tool's prompting difficulties and blind acceptance were their major concerns. Findings suggest the need for clear AI literacy training for students and a ‘human-in-the-loop’ approach in which ChatGPT feedback incorporates teachers’ guidance.