Automated Assessment of Use–Case Diagrams (AAUC), a feedback‐centric framework and tool for the automated assessment of UML use‐case diagrams, designed to support assessment practices in engineering education, is presented.
Abstract
Assessing diagrammatic artifacts such as UML use‐case diagrams is a core yet challenging task in engineering and software engineering education. In large classes, manually evaluating such diagrams is time‐consuming, subjective, and difficult to scale, particularly when student solutions exhibit structural variations and diverse labeling choices. These challenges limit the consistency of grading and the timeliness of feedback provided to students. This article presents Automated Assessment of Use–Case Diagrams (AAUC), a feedback‐centric framework and tool for the automated assessment of UML use‐case diagrams, designed to support assessment practices in engineering education. The proposed approach integrates label and structure matching, along with instructor‐configurable marking rules, within a level‐aware assessment model that supports partial credit and differentiated evaluation. Unlike approaches that focus solely on correctness detection, AAUC emphasizes the generation of meaningful formative feedback by identifying specific modeling issues, such as missing actors, incorrect relationships, and inconsistent or incomplete labeling. The framework has been evaluated using a dataset of 445 student‐generated use‐case diagrams collected over multiple academic years from undergraduate software engineering and systems analysis courses. The evaluation demonstrates that the system can robustly assess diverse student submissions, apply consistent marking policies, and generate both quantitative scores and qualitative feedback aligned with instructor expectations. By combining automated assessment with explicit feedback generation, AAUC addresses scalability challenges while supporting iterative improvement in student modeling work. The framework provides a practical, extensible solution for instructors seeking to integrate automated support into engineering education contexts involving diagrammatic design tasks.
This paper investigates the reliability of LLMs in evaluating UML diagrams generated through reverse engineering processes (source code) and asks: do LLM assessments align with those of human experts?
Olena Chebanyuk, Carles Sierra· International Conference on...· 0 citations
UML class diagrams are a crucial tool for defining the structure of software systems, but designing accurate and comprehensive class diagrams is a challenging task. Traditionally, creating UML models relies on the expertise and experience of professionals. However, with the development of AI technologies, particularly LLMs, new opportunities for software modeling have emerged. Despite this, there has been limited research on the application of LLMs in software modeling, especially in UML class diagram modeling. This study conducts experiments on several typical software system cases. Combining SDMetrics with expert manual review, this paper comprehensively evaluates the practical performance of mainstream LLMs in UML class diagram modeling from multiple dimensions, including size and completeness, relationship correctness, inheritance hierarchy, and design rule compliance. Focusing on typical defects in LLM-generated UML class diagrams, this study reveals that LLMs exhibit uncertainties analogous to human memory. Accordingly, three targeted repair strategies are proposed, including memory reinforcement, external knowledge injection, and detection-guided automated targeted repair. Experimental results obtained from the case studies indicate that (1) compared to expert-crafted class diagrams, LLM-generated UML class diagrams exhibit several issues, such as incomplete identification of key classes, confusion or omissions in relationships, insufficient or absent inheritance relationships, unused classes, and circular dependencies, and (2) after applying the repair methods, all the LLMs show varying degrees of improvement in addressing these issues. The average repair rate for key class identification reaches 85%, the coupling relationship repair rate is 46%, the inheritance relationship repair rate is 69%, while repair rates for unused classes and circular dependencies both reach 100%.
The results demonstrate that the integration of Retrieval-Augmented Generation, rubric-based evaluation, and Human-in-the-Loop validation constitutes an effective approach for AI-supported assessment of engineering laboratory reports, maintaining instructor oversight and educational integrity.
Amina Abbi, M. Skouri, M. Raoufi· International Journal of Adv...· 0 citations
Programming instructors face the challenge of providing prompt and consistent feedback, yet manual grading becomes unsustainable in large classes. While Large Language Models (LLMs) offer grading assistance, most research is based on English-language contexts and offline assessments, creating uncertainty about their dependability and the extent of human oversight needed. This study aimed to create and assess an evaluation ecosystem that integrates learning management, AI-driven task creation, LLM grading, and human review for programming courses taught in Indonesian. Employing an ADDIE-based Research and Development approach, the system was implemented for 109 students. For grading validation, instructors independently evaluated 50 assignments without access to AI predictions. The agreement was substantial, with a mean absolute error (MAE) of 4.14, a Pearson correlation of 0.986 within a 95 percent confidence interval ranging from 0.975 to 0.992, and an intraclass correlation coefficient (ICC) of 0.977. Instructor adjustments were more frequent for open-ended tasks (34.9 percent) compared to quizzes (20.2 percent). These results contributed to the development of the task-dependent human calibration (TDHC) model. The system attained a System Usability Scale (SUS) score of 88.5 and cut grading time by 87.5 percent, facilitating focused instructor review in LLM-supported programming assessments.
A. Ibrahim, Runal Rezkiawan· EDUMATIC: Jurnal Pendidikan...· 0 citations
UML (Unified Modeling Language) is a foundational tool needed for teaching software engineering. Despite this, when taught with the traditional approach, students typically find it difficult to grasp the syntax, semantics, and diagramming techniques used in UML. This paper proposes a new approach to teaching UML through an AI-enhanced gamified learning platform, which will increase student motivation, improve understanding of UML concepts, and facilitate the development of practical UML modelling skills. The proposed AI-enhanced gamified learning platform for UML includes various gamification components (e.g., XP progression, achievement badges, streaks, leaderboards, and scenario-based missions), as well as an AI-assisted diagram evaluation system and an adaptive learning system. To create a comprehensive learning experience, the platform includes (1) the use of theoretical quizzes and (2) the completion of practical diagram building assignments, an automated error detection tool, and a spaced repetition flash card system. A quantitative analysis of this gamified learning platform is conducted using a structured Likert-type questionnaire and statistical tools (IBM SPSS Statistics) on undergraduate software engineering students and Computer Science students. Quantitative results suggest that AI-assisted learning feedback and gamification have a statistically significant effect on learning effectiveness, engagement, UML skill development, and overall learner satisfaction. The results of this study provide evidence that the integration of AI-based feedback and gamified educational delivery methods can positively impact the teaching of software engineering and provide a scalable solution for delivering UML education.
Wajiha Mughal, Iqra Shahzad, Muhammad Muzzamil et al.· Pakistan journal of engineer...· 0 citations
Developing the skills required for requirements engineering students to conduct effective requirements elicitation interviews is critical yet challenging, as it requires the development of soft skills in addition to technical knowledge. Role-playing is widely adopted in requirements engineering education to support the development of these skills but is often constrained by time and resource limitations. Although recent advances in large language models (LLMs) enable automated and interactive stakeholder simulations for role-playing, their application in requirements engineering education remains limited by hallucinations and inconsistent responses. To address these limitations, this study investigates the use of retrieval-augmented generation (RAG), implemented using the LangChain framework, to support requirements elicitation activities in a requirements engineering course. We conducted controlled experiments with 69 students, comparing cohorts who interacted with non-technical faculty role-players and those who engaged with RAG-based simulated stakeholders. The results indicate that while students perceived invited stakeholders as more realistic and engaging, the RAG-based automated stakeholder produced higher-quality elicitation questions and more complete feature identification.
Manal Binkhonain, O. Alharbi· Requirements Engineering· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
