Cloud-Agnostic Graph-Based Data Management System (CAGDMS)

Abstract

The increasing reliance on cloud-based data processing has intensified challenges related to vendor lock-in, workflow fragmentation, and limited reproducibility. Data teams frequently operate across disparate platforms, resulting in dataset duplication, script re-implementation, and inefficient collaboration. This paper presents CAGDMS (Cloud-Agnostic Graph-Based Data Management System), a platform that unifies dataset management, script execution, and workflow orchestration across multiple cloud providers while maintaining comprehensive data lineage and provenance tracking. CAGDMS models datasets, scripts, and workflows as nodes and edges within a property graph, enabling fine-grained lineage tracking and reusable workflow composition. A Cloud Abstraction Layer supports cloud-agnostic execution across Azure and Amazon Web Services (AWS), automated adaptation of user-submitted scripts to distributed cloud environments, and fine-grained access control for secure collaboration. Key innovations include: 1) compiler-inspired transformation of user scripts for scalable cloud execution, 2) graph-driven workflow composition, and 3) end-to-end provenance tracking across cloud boundaries. We evaluate CAGDMS using a sixteen-script routing workflow that processes 40 GB of heterogeneous datasets across Azure and AWS. Across ten runs, CAGDMS completed the full workflow in 107.4 minutes on average, reducing end-to-end execution time by 60.2% relative to a manually coordinated baseline and by 74.7% relative to the evaluated Apache Airflow deployment. Because CAGDMS couples orchestration with script adaptation, execution-environment selection, input staging, and environment reuse, these results should be interpreted as a comparison between complete deployment configurations rather than as an isolated comparison of orchestration engines. Within the evaluated conditions (one predominantly sequential workflow, $\approx 40$ GB of data, and two cloud providers), the results demonstrate that CAGDMS can provide substantial execution-time improvements while preserving cloud-agnostic execution, automated workflow coordination, and full lineage capture. By overcoming single-ecosystem limitations, CAGDMS provides a collaborative, reproducible, and cloud-agnostic foundation for complex multi-cloud data workflows.