Skip to content
Book Open access

Beyond Syntax: Method-Compliant AI Assistance for Low-Resource Modeling Languages

Oct 2026 · Proceedings of the ACM/IEEE 29th International Conference on Model Driven Engineering Languages and Systems · 0 citations · 19 references

Abstract

Rigorous evaluation of AI-assisted modeling remains an open challenge, particularly for low-resource domain-specific languages where training data is scarce and methodological correctness matters as much as syntactic validity. We present an evaluation framework and protocol-driven architecture for this setting, instantiated on the Sierra systems engineering methodology — a DSL defined within the Ontological Modeling Language (OML), which functions as a meta-language for building ontological schemas, analogous to how frameworks like Django sit atop a general-purpose language. Our architecture keeps the LLM as an intent interpreter while routing all model manipulation through validated MCP tools that expose the same write operations the OML IDE uses, with method enforcement driven by a SHACL shape catalog that encodes Sierra’s constraints. We ground our evaluation in the publicly available FireForce VI aerospace case study, deriving tasks directly from its documentation across three difficulty levels — from granular, single-relation assertions to open-ended whole-description modeling from raw domain text. Each task is evaluated with and without tool augmentation, and with and without workspace example files, with a frontier-model baseline on the hardest task tier. We measure syntax accuracy, method compliance, and API cost, arguing that method compliance is a critical but underexplored dimension of MDE intelligence evaluation. We find that tool augmentation has the greatest impact at higher difficulty levels, supporting the hypothesis that deterministic scaffolding matters most precisely where probabilistic generation is least reliable.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.