"It's Okay Sometimes": Insights into Students' Nuanced Evaluations of their Programming Abilities While Using LLMs

Abstract

Large Language Models (LLMs) are changing how students program. Amid questions about what skills will be important in the future [6] and how LLMs will impact learning and self-efficacy [1, 3, 7, 8], we explore how introductory computing students define and assess programming ability. Prior work shows that many students believe they are performing poorly in expected programming moments, like struggling with errors, and that these beliefs correlate with low self-efficacy [4]. This reveals that many students adopt inaccurate criteria when assessing programming ability, in part due to misconceptions about the programming learning process [2, 4]. However, we do not know how LLMs are changing students’ beliefs about programming. Thus, we ask: how do introductory computing students evaluate their ability when programming with an LLM? First, we surveyed 21 students, asking free-response questions about LLM usage and how it reflects on programming ability. Through qualitative analysis, we identified nine criteria that students use to evaluate ability when programming with an LLM. Through iterative testing, we converted these criteria into 14 vignette questions in the style of [4]. For example, the criterion “cannot program a function without an LLM” was converted to “Hassan is working on a programming problem. He can’t figure out the algorithm for a function. So, Hassan asks an LLM to write the function for him,” followed by a Likert-scale question asking if the student would be doing badly in that scenario. We distributed this survey to another 31 students; responses are shown in Figure 1. Figure 1: Students’ self-assessments in response to the vignettes. Disagreement indicates a non-negative self-assessment, agreement indicates a negative self-assessment. Most students agreed that generating code without understanding it indicates low ability, but using an LLM to explain a concept does not. Unlike the previously identified self-assessment criteria [4], it is not immediately clear whether the newly-defined criteria are inaccurate. That is, should a student think they are doing badly if they use generated code without understanding it? To answer this, we must consider how LLMs do or do not support both the students’ and instructors’ learning goals. We argue that Communities of Practice (CoP), a theory stating that learning is situated within communities which shape what is learned and how [10], can help answer this question. Instructors choose learning goals and allow or disallow LLMs, creating a classroom CoP. However, students may be part of multiple computing CoPs, as they learn computing for many reasons [5, 9]. Different CoPs expose students to various learning goals, and the goals students adopt shape whether they use or refuse LLMs [8]. We believe the accuracy of self-assessment criteria thus depends on the intersections of the classroom CoP and other CoPs students are members of or aspire to join, since students’ self-assessments are grounded in the words and actions of others [2]. This conceptualization contributes to theoretical understandings of self-assessments, and suggests that we must consider how curricula can adopt or resist LLMs in support of shared learning goals.