Evaluating large language models on text-based engineering design tasks: performance gaps and failure modes

Abstract

Generative Artificial Intelligence (AI) and Large Language Models are increasingly considered for integration into engineering design workflows, yet their actual capabilities on the analytical and optimization tasks that arise in engineering design remain poorly understood. In this study, we examine Large Language Model-generated answers to 86 text-based questions covering three task categories chosen for their direct correspondence to typical engineering design activities: 50 problem-solving tasks from the Kangaroo Math Competition, 20 structural engineering design tasks involving stress analysis and cross-section selection, and 16 engineering optimization tasks involving conflicting constraints and trade-offs. Across the evaluated models, we observe that performance is consistently higher on the general Kangaroo problem-solving questions than on the engineering geometry and optimization tasks, even though these draw on a comparable set of underlying skills. The study further exposes systematic failure modes across the examined task types, including geometric reasoning, multi-step numerical computation, and optimization under conflicting constraints. These findings characterize the specific capabilities that AI-enabled design systems must develop to be reliable in practice, and establish an empirical baseline for evaluating future augmentation strategies such as tool integration and structured design workflows.