GeoOrchestra: Orchestrating Heterogeneous Geo-Distributed Training with Network-Aware Scheduling

Abstract

Scaling Large Language Models (LLMs) increasingly mandates the aggregation of geographically dispersed resources. However, cross-datacenter (Cross-DC) training faces significant hurdles due to extreme hardware heterogeneity and limited Wide-Area Network (WAN) bandwidth. Empirical measurements on a 2,000 km Cross-DC testbed reveal that the interplay of hardware mismatches, combinatorial search complexity, and shared network contention creates severe performance bottlenecks for heterogeneous training. To address these complexities, we propose GeoOrchestra, a system that decouples resource filtering from fine-grained strategy search. By abstracting compute nodes via computation and memory profiles while modeling WAN links as a virtual hard pipe, GeoOrchestra deconstructs the parallel strategy search problem into three cascading stages. Specifically, it employs hetero-aware pruning to filter invalid resource sets, followed by a resource-driven search that exploits resource disparities to maximize efficiency. Finally, the system orchestrates training traffic via dynamic time-slot slicing to enforce lower-bound bandwidth guarantees for individual jobs. Evaluations on our testbed demonstrate that GeoOrchestra achieves up to 1.6× to 1.8× throughput improvement over state-of-the-art baselines, while maintaining stable execution under real-world shared WAN contention.