Evaluating LLM-Assisted Multiple-Choice Question Generation in Requirements Engineering: Prompting Strategies and Professor Validation

Abstract

Large Language Models (LLMs) are increasingly being explored for their potential to automate the generation of multiple-choice questions (MCQs). However, the quality of these questions is still influenced by various factors, including pedagogical, conceptual, and linguistic considerations. This article presents the outcomes of a new experimental round conducted in 2026, focusing on MCQ generation within the Requirements Engineering domain, and builds upon a previous study published in 2025. The current study uses the same domain, Prompt Engineering techniques, evaluation criteria, and validation protocol, and merges automatic evaluations by LLMs with assessments from professors. In total, 200 questions were generated and analyzed. The results indicate that automatic evaluations and human assessments often produce differing judgments, with LLMs generally assigning higher scores than their human counterparts. Additionally, the findings reveal that Prompt Engineering techniques and generator models exhibit different behaviors across evaluators. Challenges related to the criteria of diversity and complexity persist in automated question generation. This study highlights the considerable potential of LLMs to enhance assessment design, while also emphasizing the essential role of human curation and the need for continuous validation in the rapidly evolving landscape of LLM ecosystems.