SysName, a production-oriented pipeline that automates device configuration end-to-end for Modbus RTU, OPC-UA, Profibus DP, and CANopen, builds a hybrid dense-sparse retrieval index augmented by an ontology graph derived from ECLASS, AAS, and SOSA/SSN, using a BGE-M3 encoder with a cross-encoder reranker to surface relevant manual passages.
Abstract
Industrial device commissioning requires engineers to manually extract hundreds of protocol-specific parameters from heterogeneous PDF manuals and transcribe them into supervisory control systems, a time-intensive, error-prone workflow. This paper presents SysName, a production-oriented pipeline that automates device configuration end-to-end for Modbus RTU, OPC-UA, Profibus DP, and CANopen. It builds a hybrid dense-sparse retrieval index augmented by an ontology graph derived from ECLASS, AAS, and SOSA/SSN, using a BGE-M3 encoder with a cross-encoder reranker to surface relevant manual passages. A local LLM (T=0.1) generates ontology-aligned JSON-LD configurations via protocol-specific prompts and a four-step repair pipeline. A two-stage abstention gate, combining a reranker-score threshold and an IRI resolution ratio, blocks unsafe LLM invocations and filters low-coverage configurations before SHACL validation. On a gold set of 28 field-level queries, the hybrid retriever reaches 0.96 HitRate@10, and the reranker raises MRR@10 from 0.56 to 0.63 with perfect score separation for abstention. The generator attains field-level F1=0.87 with exact match on 9 of 12 runs. End-to-end runs on an H100 GPU complete in 2.6-6.6s per device with zero unsafe writes and zero silent failures on a five-device benchmark; every unsuccessful run is flagged by abstention or deployment verification. Component-wise evaluation localises the single systematic failure to OPC-UA generation, invisible to end-to-end metrics alone. A case study commissions a physics-simulated Universal Robots UR5e robot from unmodified vendor documentation (254-page manual, 8-page register list, 496 chunks), reaching field-level F1=1.0 over three runs with read-back and joint-consistency verification. An ablation study and comparison with five industrial-LLM systems complete the analysis.
The configuration of AUTOSAR Basic Software (BSW) modules relies on verbose AUTOSAR XML (ARXML) files that are complex, error-prone, and costly to produce manually—particularly for safety-critical modules governed by ISO 26262. This paper presents a two-stage approach to automating BSW configuration generation that generalizes to any ECU Configuration (ECUC)-based module: a fine-tuned large language model (LLM) generates a compact JSON intermediate representation capturing only semantically meaningful parameters, which a deterministic expansion function reconstructs into schema-conformant ARXML. We fine-tune three open-weight models (Qwen3-8B, Ministral-3-8B-Instruct, Llama 3.1 8B) with Quantized Low-Rank Adaptation (QLoRA) on 6050 compositionally generated Watchdog Manager (WdgM) samples spanning five complexity tiers with 30+ prompt templates, and introduce a hierarchical evaluation pipeline combining schema validation with referential integrity, structural completeness, parameter accuracy, and semantic constraint satisfaction. The compact representation reduces output tokens by approximately 8–10× compared to full ARXML. All three models achieve closely comparable performance (0.815–0.836 overall score), with Llama 3.1 8B scoring highest (0.836) and every model reaching ≥93% schema validity and ≥72% parameter accuracy—an 8.4× improvement over zero-shot baselines. Decomposing generation into LLM-driven semantic capture and deterministic expansion is an effective strategy for verbose, schema-governed configuration formats, extensible to other AUTOSAR modules beyond WdgM.
Amr Samy, Ahmed Moro, M. Taher· Applied Sciences· 0 citations
Enterprise compliance management requires rapid adaptation to evolving regulatory frameworks (e.g., DORA, AI RMF, FedRAMP) and tight remediation SLAs. Traditional static orchestrators often fail in hybrid cloud environments where event-driven assessments demand that automation code adapt to runtime context in seconds. This paper presents lessons learned from evaluating six large language models for AI-driven workflow generation in a production enterprise platform, benchmarked across 29 real-world IT automation scenarios, two generation pipeline architectures, and eight independent runs per prompt-model-pipeline configuration (2,784 runs total). Our initial pipeline used monolithic workflow generation, achieving 31.5-82.8% structural success rates (JSON schema validity and correct UI rendering), with most models struggling on complex JSON generation. We developed a redesigned piecewise pipeline that decomposes workflow construction into variable scaffolding, base block assembly, and nested block generation, raising structural success to 74.1-97.8% across all models. We analyze production tradeoffs including cost (USD 0.008-0.20 per workflow), latency (under 50s for interactive use), and model selection. Piecewise decomposition enables smaller models (e.g., mistral-small at 95.7% structural success and USD 0.01 per workflow) to reach production viability, removing dependency on expensive frontier models. While mistral-medium-2505 and gpt-oss-120b achieved the highest structural success (96.1% and 97.8%), mistral-medium-2505 carries a 19x cost premium versus mistral-small. Our deployment lessons highlight the need to separate structural validity from semantic correctness (logical fulfillment of user intent) and provide a solution for model-agnostic, scalable automation in cloud engineering.
Xavier Wrenn, Radoslav Raykov, Aleksandar Angelov et al.· 0 citations
Large language models (LLMs) and LLM-based agents are increasingly being deployed to automate complex workflows, promising to revolutionize data management and processing. However, existing benchmarks predominantly focus on simplified Text-to-SQL translation or data analysis, leaving the critical and complex domain of end-to-end data engineering largely unexplored. To bridge this gap, we introduce DataClawEval, the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios. Built upon production-grade code authored by professional enterprise data engineers, it comprises 100 rigorous, end-to-end tasks spanning five execution engines: PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL. Rather than non-deterministic LLM-as-a-judge scoring, each task is executed within a case-specific, isolated sandbox and graded by deterministic, rule-based scripts. Evaluating 16 frontier agents exposes critical limitations: The strongest model attains only 74.9 overall, and no single model dominates, as each excels on a different engine, revealing strict domain specialization rather than omnipotent proficiency. Thus, autonomous data engineering remains a formidable, unresolved challenge. We release our dataset, containerized environments, and deterministic evaluation scripts at https://github.com/Dicemy/DataClawEval/tree/master
Debin Meng, Jiaming Yang, Zefang Zong et al.· 0 citations
Industrial technical reports contain high-value knowledge for maintenance, troubleshooting, and product engineering, but their heterogeneous structure (dense prose, specifications, tables) makes them difficult to index and reason over with standard retrieval and QA pipelines, and no public instruction-tuning or benchmark datasets are built from such documents. We address this gap with Industrial-Instruction, contributing (i) two open QA datasets built from real industrial technical reports and (ii) the end-to-end pipeline that produces them. Using 906 public Panasonic documents (7,525 pages), we apply layout-aware extraction, build a semantic retrieval index, and synthesize multiple-choice QA grounded in retrieved evidence under five query-document relationships (irrelevant retrieval, single-/multi-document support, single-/multi-document answer). After filtering an initial 23.9k generated samples, each dataset provides approximately 13.6k QA pairs with source documents and a held-out benchmark split. Fine-tuning small open LLMs (under 10B parameters) improves Set-Match Accuracy from 28.5% to 42.0% and F1 from 46.6% to 63.5% on the Panasonic benchmark. We release two parallel versions built by the same pipeline: one generated with the open-weight Qwen3-30B-A3B-Instruct model and one with the closed, API-based Claude-Opus-4.6 model, enabling a direct comparison of open- versus frontier-model data generation. The Claude-Opus-4.6 dataset yields a cleaner raw corpus and larger fine-tuning gains, at roughly two orders of magnitude higher cost. MMLU evaluation shows models trained on the Claude-Opus-4.6 data retain essentially all general knowledge, versus a small but measurable forgetting effect for the Qwen-generated data. Together, these datasets and pipeline offer a practical, reproducible path toward scalable industrial benchmarks and training data from real-world documentation.
Parsa Bakhtiari, Hassan Bashiri, Alireza Khalilipour et al.· 0 citations
Large Language Models (LLMs) are increasingly used to generate executable software environments from repository artifacts. However, functional executability does not necessarily imply conformity with architectural, security, workflow, and production intent. This study investigates whether a frontier LLM can generate Dockerfiles and Docker Compose configurations for multi-service applications using repository contents without access to developer-authored deployment artifacts. Three heterogeneous repositories combining Python, Node.js, .NET, React, Rust, Java, Redis, PostgreSQL, and MySQL-compatible infrastructure were evaluated using deterministic end-to-end HTTP oracles and manual structural comparison. All three generated environments became functionally operational, although one required a Rust base-image update from version 1.85 to 1.88. The model correctly reconstructed service topology, application ports, infrastructure dependencies, service hostnames, a background worker, hidden proxy configurations, and a file-based Docker secrets mechanism. However, it consistently omitted network segmentation, multi-stage builds, dependency-layer caching, live-reload volumes, production frontend serving, restrictive backend-port policies, and cross-platform build logic. Based on these observations, the study formalizes the distinction between functional correctness and deployment-intent fidelity and analytically derives a minimal explicit deployment specification for information that cannot be reliably inferred from repository artifacts.
Oleg Grynets, Kyrylo Fursov, V. Lyashkevych et al.· 0 citations
Modern FPGA computer-aided design (CAD) flows have grown increasingly complex, integrating numerous stages, configuration parameters, timing constraints, and physical implementation specifications. As designs scale, failures often arise from subtle interactions across command-line options and constraint files, making debugging time-consuming and heavily dependent on expert knowledge. Identifying the root cause of such failures and determining the appropriate corrective action remains a major productivity bottleneck in CAD workflows. This paper presents VTR-LLM, a fully automated, multi-agent framework for diagnosing and resolving failures in the Verilog-to-Routing (VTR) CAD flow. VTR-LLM leverages large language models (LLMs) in combination with retrieval-augmented generation (RAG) and specialized agents that target distinct sources of errors, including command-line invocations, timing constraints (i.e., Synopsys Design Constraints or SDC), and floorplanning specifications. A Classification Agent dynamically assigns each failure to the most appropriate agent and supports sequential resolution for compound failures involving multiple error sources. The system leverages LLMs without requiring fine-tuning, enabling use of the latest models such as GPT-OSS-120B, by using RAG and intelligent agents to add domain-specific context and behaviours. We evaluate VTR-LLM using a dataset of 92 distinct VTR failure cases spanning multiple error patterns and levels of complexity. VTR-LLM can resolve 95% of all failures fully automatically, with robust performance across error categories. Additional studies demonstrate the impact of documentation retrieval scope, tool iteration, and LLM model choice on resolution accuracy and inference cost.
Mohamed A. Elgammal, Jamie Wu, Lynne Liu et al.· ACM Transactions on Reconfig...· 0 citations