Efficient Distributed Mini-Batch GNN Training With Decentralized Batch Processing

Abstract

Distributed mini-batch Graph Neural Network (GNN) training is a crucial technique for learning large-scale graphs. Existing training methods typically rely on a centralized batch processing pipeline, where a single worker manages the entire lifespan of a batch. Consequently, the communication volume is linearly proportional to the computation graph size, which grows exponentially with the number of GNN layers, leading to prohibitive communication costs. In this paper, we propose DEGNN, an efficient distributed GNN training framework that adopts a push computation to data strategy. By distributing the neighbor preparation workload to workers where the graph data resides, we design a decentralized batch processing pipeline in DEGNN that decouples the communication volume from the computation graph size. To realize this pipeline effectively, we propose LAMP, a Locality-Aware Multi-layer neighbor Preparation strategy. Specifically, LAMP constructs the aggregation skeleton via a locality-aware Computation Graph (CG) generation strategy. It then performs multi-layer aggregation to capture multi-hop neighborhood information without cross-worker communication, leveraging quantized historical embeddings to satisfy the recursive dependency of GNN layers. To further enhance end-to-end efficiency, we introduce a two-level computation graph fusion mechanism and an asynchronous pipeline to minimize computation overhead and hide communication latency. Empirical results demonstrate that DEGNN outperforms state-of-the-art distributed training systems, achieving up to 6.05× speedup while maintaining model accuracy.