Evaluating AI-generated curriculum designs in vocational education: Evidence from a No-AI diagnostic assessment
TL;DR
It is suggested that no-AI diagnostic assessment can make students’ judgement of AI-generated vocational work visible and can help teachers identify where AI-assisted task completion has not yet developed into stable professional judgement.
Abstract
This study examines how vocational students evaluate AI-generated curriculum designs after using generative artificial intelligence to support course-planning tasks. It reports an interpretive single-case study of a Study Tour Curriculum Design course in a Chinese vocational college. Data comprised course task briefs, the instructor’s reflective teaching memo, and a closed-book handwritten diagnostic assessment completed by 39 second-year students without AI access. The diagnostic assessment yielded 182 substantive diagnostic claims, which were analysed through thematic and directed content analysis, with students’ complete responses also assigned holistic written judgement levels. Students could state broad criteria for judging AI-generated work, including accuracy, feasibility, learner appropriateness, safety, and the need for human revision. Their applied diagnostic performance was uneven. Most identified visible implementation problems such as learner mismatch, time pressure, site overload, and safety insufficiency, while fewer diagnosed deeper issues of curriculum logic, assessment operationalisability, fact verification, and author responsibility. The study identifies this pattern as a stated-applied judgement gap. The findings suggest that no-AI diagnostic assessment can make students’ judgement of AI-generated vocational work visible and can help teachers identify where AI-assisted task completion has not yet developed into stable professional judgement. The study offers a transparent classroom case of how AI-generated outputs can be evaluated in vocational education, while recognising the limits of a single-case design.