VADER: Validation-Driven DSL Generation with Small Language Models

Abstract

Model-driven and low-code platforms rely heavily on domain-specific languages (DSLs) to capture structured system intent. While large language models (LLMs) have shown impressive capabilities in generating DSLs from natural language descriptions, proprietary models such as GPT-5 remain expensive, opaque, and often brittle in practice. This paper explores whether carefully orchestrated small language models (SLMs) can achieve comparable modeling quality while offering greater transparency, affordability, and control. We introduce a modular multi-agent framework in which small models take on specialized roles-including intent classification, DSL generation, syntactic validation, consistency checking, and automated repair. The framework supports two generation pipelines: one producing DSL code validated directly against a Lark grammar, and another generating structured JSON validated through Pydantic schemas. To address the inherent limitations of small models, we incorporate a lightweight prompt expansion step that enriches minimal user inputs with relevant domain concepts before DSL generation. We evaluated the approach across four experiments involving 30 different open-source SLMs. We measured syntactic correctness, structural similarity to GPT-5 outputs, alignment with expert-written DSLs, and overall modeling quality using independent LLM-as-a-judge evaluations. The results are compelling: feedback-driven repair dramatically increased the number of models producing valid DSLs (from 9 to 23), while prompt expansion improved structural similarity by 20-40 points and boosted F1 scores against expert references across all models. Blind evaluations by LLM judges consistently preferred the outputs from the expanded and repaired pipelines. Our findings demonstrate that architectural choices-such as prompt expansion, validation-feedback loops, and role-specialized agents-matter far more than raw model scale. This work shows that high-quality DSL generation is achievable with small, transparent models, opening the door to more accessible, costeffective, and trustworthy model-driven development platforms.