Evaluating the Ability of ChatGPT as an AI-Powered Chatbot: Assisting EFL Students' English Scientific Paper Writing

Abstract

ChatGPT is an AI-powered chatbot capable of generating human-like text based on the input provided through conversations like humans in natural language. This study aims to evaluate the capability of ChatGPT as an AI-powered chatbot in assisting EFL students in English scientific paper writing, with a particular focus on undergraduate theses. This research applies a content analysis approach to examine the textual outputs generated by ChatGPT in response to academic thesis-related prompts. The researcher conducted 13 trials by inputting various thesis-related prompts into ChatGPT. The queries covered multiple components of a thesis, including the title, background (research gap, research question, and significance), literature review, methodology (design, data collection, and analysis), findings, discussion, conclusion, and references. The results indicate that ChatGPT was able to generate coherent responses for most thesis components. However, its output for the findings, discussion, and conclusion was inadequate due to the absence of actual research data and context, which are essential for producing valid results. Additionally, when given a differently formulated research objective, ChatGPT occasionally provided inconsistent or inappropriate responses. While the tool demonstrates a strong contextual understanding of text and generates relevant content, its limitations highlight potential issues related to academic integrity, including plagiarism and ethical concerns. This study emphasizes that while ChatGPT can effectively support various stages of English scientific paper writing, its outputs require careful human evaluation, verification, and refinement before being incorporated into academic work.