Best Practices for Assessing Students’ Programming Skills in the Era of Large Language Models

Abstract

Programming skills assessments refer to the conceptual and procedural evaluations of students’ programming skills [1, 13]. Mandinach and Linn [7] defined conceptual and procedural skills as the students' abilities to combine programming language features (i.e., syntactic knowledge) and templates (i.e., conceptual knowledge) into a working computer program. In other words, conceptual knowledge refers to the relationships between items of knowledge, allowing students to identify the links between and among these pieces of knowledge [7]. If students can make these connections, they are said to have conceptual understanding. Procedural knowledge, on the other hand, is the actionable term where students apply their knowledge [9]. Assessing students’ programming skills may include take-home coding problems, exams, or quizzes [4, 12]. However, these methods are prone to plagiarism or cheating in both offline [4] and purely online learning environments [10]. These problems have worsened with the emergence of large language models (LLMs) for two reasons. First, LLMs can answer test items and generate code with high accuracy [2, 3, 11]. This accuracy can lead to their use being perceived as cheating. Second, LLMs can be utilized as a programming aid where they are capable of generating code [2, 6]. Thus, the emergence and spread of LLMs has blurred the line between cheating and facilitated learning in programming skills assessments [5]. In this current educational context, there are growing concerns regarding the validity of programming assessments. It is no longer a question of whether LLMs should be adopted as educational tools since they are increasingly recognized that students will inevitably utilize this technology. Thus, there is a pressing need to redefine or redesign programming skills assessments to align with the current educational settings shaped by LLMs. One study has suggested solutions such as coding interviews, collaborative coding challenges, project-based assessments, and oral examinations [8]. This current study intends to extend the literature in several ways. First, it is important to determine whether teachers have redesigned their programming assessment tools. Second, if they have, it is necessary to identify the principles that programming teachers follow in designing their assessments. Third, there is a need to document which redesigned programming assessment tools work well and which do not. Practices that do not work are just as important as those that do, as they offer valuable lessons for other teachers. Finally, propose an assessment framework. To answer these objectives, this study uses the Delphi technique. This approach builds programming teachers’ consensus through iterative rounds of surveys. In this study, it will be used to determine best practices in programming assessments. Respondents will be college-level programming teachers. The study aims to provide guidelines on programming assessments that could be beneficial for face-to-face, online, and lifelong learning environments.