From Informal to Conformant Models: Benchmarking Vision--Language Models for UML Generation

Abstract

Software teams routinely sketch and draw informal diagrams to capture design intent, yet these artifacts rarely evolve into conformant models that automated tooling can process or that can support later stages of development. This tension motivates flexible modeling, which aims to bridge free-form sketching and canonical modeling by enabling progressive formalization. Recent advances in vision–language models suggest a new mechanism for operationalizing this bridge by translating informal visuals into canonical modeling artifacts, but evidence on current capabilities remains limited. In this paper, we provide a systematic empirical benchmark of cross-diagram, cross-model performance of four vision–language models: Claude 3.7 Sonnet, GPT-4o, Llama 4 Maverick, and Gemini 2.5 Pro, for generating five UML diagram types: class, sequence, activity, use case, and state machine. We used a dataset of informal and schematic diagrams extracted from scientific publications and evaluated outputs using automated syntactic validation and structured manual semantic assessment of element and relationship correspondence to the source. Our results showed that most models produced syntactically valid diagrams with high accuracy, particularly for sequence and use case diagrams, while semantic fidelity remained substantially lower and varied across models and diagram types. Activity diagrams were easiest to generate semantically, whereas use case diagrams were most challenging due to conceptual ambiguity, highlighting a stable syntax–semantics gap that currently limits fully automated design formalization.