Towards Model-Driven Generation of Domain-Specific Modelling Agents for Local Language Models

Abstract

Current frontier LLMs leverage vast training datasets with knowledge about a broad range of programming languages and frameworks, and can often solve tasks with minimal prompting. However, there are scenarios where remote LLMs are unsuitable, due to cost, privacy, security, or intellectual property concerns, and local language models need to be used. These smaller local language models typically require more careful guidance in order to achieve acceptable performance. In this paper, we evaluate the impact of using local models (in the 14-31B range) for object-oriented modelling, and we propose an approach for partially automating the creation of agents that "think in code" to solve such problems, instead of directly producing an output. We compare the direct invocation of Claude Sonnet 4.6 with that of several open-weight models, and with the use of those open-weight models while "thinking in code" as proposed. We observe that the best open-weight models, when "thinking in code", can produce models with similar Jaccard similarity scores to the model solutions in the Golden UML ModelSet dataset as Sonnet, although at the cost of increased token consumption. Based on the observed potential for generating domain-specific agents that reduce the demands on the LLM and therefore on the hardware requirements, we set out several lines of further work.