Reliable isomorphic physics problem generation with large language models
TL;DR
An AI-powered system for generating isomorphic physics problems using large language model (LLM)-based agent workflows that combines prompt chaining, agent-based verification, and automated LaTeX compilation within a publicly accessible website hosted on a Raspberry Pi is presented.
Abstract
This study presents an AI-powered system for generating isomorphic physics problems using large language model (LLM)-based agent workflows. The system is designed around three practical goals: preserving the same conceptual and problem-solving structure as the original problems, varying construct-irrelevant features such as scenarios and numerical values, and producing questions that are directly usable without expert revision. The workflow combines prompt chaining, agent-based verification, and automated LaTeX compilation within a publicly accessible website hosted on a Raspberry Pi. To evaluate the system, we developed an eight-item rubric and tested the system using 13 multiple-choice questions from a calculus-based introductory Newtonian mechanics course. The evaluation results showed that 89% of the questions generated were rated as fully specified and directly usable. However, the system also showed limitations. The results suggest that LLM-based systems have significant potential for reliable instructional problem generation while also highlighting important challenges for future development.