Agentic AI-Driven Data Engineering: A Framework for Self-Directed Enterprise Data Pipelines

Abstract

Enterprise data pipelines have grown in scale and interdependency faster than the operational practices used to maintain them. Pipeline failures are still predominantly diagnosed and remediated by human engineers reading logs, tracing lineage, and applying tacit knowledge, which produces long mean-time-to-recovery and concentrates institutional knowledge in a small number of individuals. This paper proposes an agentic architecture in which large language model (LLM) agents, constrained by explicit tool interfaces and policy guardrails, perform continuous monitoring, root-cause triage, and bounded remediation of production data pipelines. The framework specifies five cooperating agent roles — Observer, Diagnostician, Remediator, Verifier, and Orchestrator — operating over a shared state store containing pipeline metadata, lineage graphs, data-quality expectations, and incident history. We define a capability-tiering model that maps each remediation action to one of four autonomy levels (observe, recommend, act-with-approval, act-autonomously), and we describe a human-in-the-loop governance layer providing approval routing, immutable action logging, and rollback. We further propose an evaluation protocol covering task success rate, mean-time-to-detection, mean-time-to-recovery, false-remediation rate, and escalation precision, together with a reference workload derived from common ingestion, transformation, and schema-evolution failure modes. The paper concludes by analysing failure modes specific to agentic operation — hallucinated root causes, unsafe tool invocation, reward-free drift, and accountability gaps — and argues that autonomy must be earned incrementally through measured reliability rather than granted by architectural default.