Hand-drawn sketch to parametric 3D CAD using a multimodal LLM and an intermediate representation
Abstract
Informal hand-drawn sketches are used in mechanical design to convey information about part geometry, but their translation into parametric 3D CAD models is not fully addressed. Multimodal Large Language Models (LLMs) can interpret visual information and generate executable macros, opening the possibility to support a drawing-to-parametric CAD approach with minimal user involvement. This study proposes a two-step workflow based on an explicit and traceable interpretation process. First, an LLM interprets isometric sketches and generates, using a fixed prompt, a predefined Intermediate Representation (IR). The IR classifies the extracted information as: Visible, Inferred, Assumed or Missing, and lists assumptions. Human validated IRs are then converted into CATIA V5 macros using a second prompt. The evaluation was conducted on 41 sketches, including prismatic blocks, cylinders, combinations of these primitives, and solids with one or multiple cylindrical holes. In the first stage, the IR-based interpretation achieved: 100% schema validity, 90.2% base type identification, 82.9% correct composite count identification, 78.1% feature count correctness and 65.9% assumptions correctness. For the second stage – the macro generation phase, 14 IRs were selected from the 27 IRs correctly interpreted in the first stage, as several corresponded to similar IR outcomes. The macro generation success rate was 92.9%, execution success rate was 78.6% and structural correctness rate was 42.9%. Failure analysis showed that IR errors were mainly related to ambiguous cylindrical features, while macro errors were associated with feature orientation and geometric references. Separating interpretation from CAD generation provided the advantage of a clearer identification of failure modes and supported a more controlled and transparent Sketch-to-CAD pipeline. The study was exploratory and the investigated sketches were limited to predefined geometric classes. Therefore, the results should not be generalized to more complex mechanical parts or CAD modeling scenarios.