A PROTOTYPE SYSTEM FOR LLM-BASED AUTOMATIC GENERATION OF MULTIPLE-CHOICE QUESTIONS IN MACHINE LEARNING PROGRAMMING

Abstract

In recent years, the importance of mathematical, data science, and AI education has grown significantly, making it necessary to learn machine learning programming. In programming education, while acquiring knowledge through lectures is important, the experience of actually creating programs helps solidify that knowledge. However, preparing numerous programming assignments places a significant burden on educators. This study proposes a system that utilizes large language models (LLMs) to generate programming tasks for scikit-learn, a machine learning library written in Python. It describes the implementation and evaluation of a prototype version of this system. The programming exercises used in that system implement a multiple-choice question format where part of the program is left blank, and the student must fill in the missing text. This serves as an alternative implementation for programming exercises. Our preliminary evaluation confirmed that by refining the prompt, tasks suitable for the purpose are automatically generated. However, to achieve accurate task generation, additional instructions to the prompt are required for each target problem category. Therefore, establishing a universal prompt generation method for this purpose is a future challenge