Data-Parallel and Heterogeneity-Aware Scheduling for Geo-Distributed Serverless Scientific Workflows

Abstract

Serverless computing has emerged as a promising paradigm for executing scientific workflows characterized by complex task dependencies, data-intensive operations, and high computational demands. However, most existing scheduling approaches assume homogeneous, single-region environments and primarily focus on isolated function execution. These assumptions overlook two critical challenges: (i) the inherent data-parallel nature of workflow tasks, and (ii) the heterogeneity of computing resources across geo-distributed serverless platforms. In this paper, we address these limitations by proposing a novel scheduling framework for geo-distributed serverless environments that explicitly models intra-function data parallelism, heterogeneous abstract resources, and regional concurrency constraints. We formulate a makespan minimization problem in which each function can either execute entirely on a single high-capacity resource or be partitioned across multiple heterogeneous lower-capacity resources, subject to region-specific concurrency limits.To solve this problem, we design a Deep Q-Network (DQN)-based scheduler augmented with two auxiliary heuristics. The first, Critical Workload First, prioritizes high-workload functions through an exhaustive split-deployment search over heterogeneous abstract serverless resources. The second, Load-Aware Heuristic, selects execution regions using a weighted load metric combined with penalty-based resource assignment. We evaluate our approach on five representative scientific workflows BWA, Montage, Inspiral, CyberShake, and SIPHT and using real-world round-trip time measurements from Azure Function deployments across three geo-distributed regions. Experimental results demonstrate that our DQN-based scheduler reduces makespan by up to 29.13% compared to state-of-the-art approaches.