Computing Resource-Aware Operation Optimization Strategy for MPI Jobs in Cloud-Native Environment
TL;DR
A resource-aware optimization framework that dynamically selects the MPI process count and performs node- and NUMA-aware process placement and reduces task-sequence execution time and improves the evaluated resource-utilization metrics by more than 30%.
Abstract
Large-scale high-performance computing workloads in petroleum geophysical exploration commonly use the Message Passing Interface (MPI) as their parallel programming model. However, MPI does not provide native mechanisms for resource management, which complicates the efficient execution of multiple MPI jobs in shared-resource environments. As MPI workloads migrate to cloud-native platforms, their high degrees of parallelism and communication-intensive behavior may lead to scheduling delays and contention for shared resources, resulting in prolonged execution time and inefficient resource utilization. This paper identifies two major limitations of existing cloud-native MPI deployments: cluster-unaware process-count selection and insufficient exploitation of Non-Uniform Memory Access (NUMA) locality. To address these limitations, we propose a resource-aware optimization framework that dynamically selects the MPI process count and performs node- and NUMA-aware process placement. Experimental results show that the proposed parallelism-selection method reduces task-sequence execution time by at least 18% and improves the evaluated resource-utilization metrics by more than 30%. The topology-aware placement method further reduces execution time by at least 10% compared with the evaluated affinity baselines under the tested shared-resource cluster configurations.