Agentic Data Services: A Control-Plane Architecture for Adaptive Data Workflows

Abstract

Modern data platforms rely on pipeline-oriented architectures that are rigid, hard to adapt, and lack native auditability. We present Agentic Data Services, a control-planedriven architecture for Big Data as a Service (BDaaS) that models workflows as adaptive, policy-aware service entities rather than static directed acyclic graphs (DAGs). The architecture combines (i) a dual-record execution model adapted from pharmaceutical batch manufacturing-versioned Master Batch Records (MBRs) for workflow definition and immutable Electronic Batch Records (EBRs) for execution traces-and (ii) workflow-level semantic caching that reuses results across semantically similar requests. We implement the system as Agentic DataHub, a set of Rustbased microservices deployed on Kubernetes, and evaluate the semantic-caching component on a reproducible benchmark using sentence-transformer embeddings (all-MiniLM-L6-v2) and a FAISS flat inner-product index. For clustered workloads-semantically related requests grouped into 10 clusters with 70% intra-cluster similarity-the cache reduces backend requests by 81% and median latency by 93%, with 40% P95 latency reduction. We discuss generalization across domains and the architectural constraints that bound these results.