Generative AI and Agentic Orchestration for Autonomous Data Engineering in Multi-Domain Enterprise Analytics Platforms
TL;DR
A novel Agentic Data Engineering Orchestration Framework (ADEOF) is presented that positions large language model (LLM)-powered agents as autonomous actors across the SDLC, coordinating requirements gathering from functional leaders and subject matter experts, auto-generating ETL technical design documents, and autonomously executing data quality validation and gap analysis workflows.
Abstract
Enterprise data engineering platforms operating across banking, insurance, finance, and manufacturing domains face escalating complexity in managing heterogeneous data pipelines, dimensional data models, and governance workflows that demand expert human oversight at every stage of the software development life cycle. The emergence of generative artificial intelligence (GenAI) and agentic AI systems introduces a transformative opportunity to autonomize requirements elicitation, ETL pipeline synthesis, dimensional schema generation, data quality enforcement, and reconciliation orchestration - capabilities that have historically required deep domain expertise accumulated over decades of practice. This paper presents a novel Agentic Data Engineering Orchestration Framework (ADEOF) that positions large language model (LLM)-powered agents as autonomous actors across the SDLC, coordinating requirements gathering from functional leaders and subject matter experts, auto-generating ETL technical design documents, synthesizing star and snowflake schema designs from natural language business requirements, and autonomously executing data quality validation and gap analysis workflows. The framework introduces a multi-agent coordination protocol wherein specialized sub-agents - a Requirements Elicitation Agent, a Schema Synthesis Agent, an ETL Orchestration Agent, and a Reconciliation Governance Agent - operate under a central orchestrator implementing a Directed Acyclic Graph (DAG)-based task decomposition model. Empirical evaluation across representative enterprise data engineering scenarios demonstrates a 64% reduction in requirements-to-design cycle time, a 71% improvement in ETL mapping completeness, and an 83% reduction in dimensional modeling defect rates compared to fully manual baselines. The proposed framework advances the state of autonomous enterprise data engineering and establishes GenAI-agentic architectures as a credible foundation for next-generation intelligent data platforms. Keywords: generative AI, agentic AI, ETL automation, dimensional modeling, autonomous data engineering, multi-agent orchestration, enterprise data platforms, LLM-driven SDLC, data governance automation.