Experimental evaluations demonstrate the effectiveness of the framework ASTRA, which achieves lower time-to-accuracy than a resource-heterogeneity-aware baseline and several compression-based frameworks, while preserving convergence quality and robustness across heterogeneous hybrid cloud environments.
Abstract
The rapid growth of large datasets and AI models has necessitated scalable and powerful computing resources, driving the extension of training workloads to the cloud to balance cost and performance. Among distributed learning strategies, synchronous data-parallel training remains the most widely adopted approach due to its simplicity and stable convergence. However, heterogeneity between worker nodes and unreliability of network bandwidth pose substantial challenges. Since workers must synchronize gradients at every update step, even a single straggler can significantly degrade overall performance. In this paper, we present ASTRA, an auto-adaptive synchronous training framework tailored for hybrid cloud environments. ASTRA addresses these issues with three coordinated mechanisms: i) a network-topology detection module that clusters workers into groups by pairwise connection cost to enable hierarchical aggregation, where gradients are first aggregated within each group and then across groups, mitigating slow-link bottlenecks; ii) a load balancing module to efficiently distribute workloads among heterogeneous GPUs over time, reducing resource imbalance and straggler effects; and iii) an adaptive gradient compression module that dynamically adjusts compression levels based on network conditions to minimize inter-group communication overhead. Through automated adaptation to diverse system environments, our unified approach alleviates the burden of manual configuration for AI practitioners. Experimental evaluations demonstrate the effectiveness of the framework ASTRA, which achieves lower time-to-accuracy than a resource-heterogeneity-aware baseline and several compression-based frameworks, while preserving convergence quality and robustness across heterogeneous hybrid cloud environments.
The symbiotic scaling of artificial intelligence models and high-performance computing systems continually creates algorithmic challenges in their convergence. Foundation models (FMs) are a crucial example, requiring months-long training on thousands of cutting-edge GPUs. Sharded data parallelism (DP) is the dominant strategy to accelerate such computations by splitting data and models across multiple GPUs. However, it incurs prohibitive communication overhead when deployed at scale, particularly on multi-tier interconnects with heterogeneous performance. Inspired by the efficient communication principles of federated learning (FL), this work introduces two hybrid algorithms - FL+FSDP and FL+HSDP - interleaving sharded DP with FedAvg-style aggregations. Such approaches decouple large DP deployments into smaller, loosely-coupled federation groups, requiring minimal inter-group traffic while keeping the global batch size bounded by the groups'size. Formal analysis of communication costs and experimental validation prove their scalability and flexibility. A Llama3.1 8B pre-training on 512 A100 GPUs shows that, under identical hyperparameters, FL+FSDP and FL+HSDP achieve up to 8.04 faster data processing and 4.48 lower evaluation perplexity than their counterparts, demonstrating superior computational efficiency and improved model quality. These properties stem from reduced communication overhead and the bounded growth of the global batch size relative to the federation group size.
Gianluca Mittone, Marco Aldinucci· European Conference on Paral...· 0 citations
With the rapid advancement of deep learning technology, shared GPU clusters receive an increasing number of deep learning training (DLT) jobs. Yet resource fragmentation make such clusters underutilized and forces the DLT jobs running on them to endure long turnaround times. Extensive research has been devoted to quantifying fragmentation and developing scheduling algorithms that alleviate its impact. However, existing fragmentation measures break down in the absence of workload distribution information, while current schedulers cannot continuously maintain resource fragmentation at a low level. To tackle these problems, we first introduce Scheduler-Induced Fragmentation (SIF), a metric built on the notion of partial-nodes that is independent of historical workload knowledge. We then propose COMPASS-ABS, which employs the COMPact-ASSured (COMPASS) algorithm to confine the cluster state within a tight Anchor-Based Space (ABS), whose construction fully leverages the topological alignment between dominant workload size and node capacity. Moreover. We also prove that it ensures SIF is bounded by $\frac{2}{N}$ under a workload composition condition that matches both theory and production. Evaluations implemented on a physical cluster and a simulated cluster demonstrate COMPASS-ABS effectiveness at improving resource utilization, reducing DLT job completion time by reducing fragmentation.
The shift of deep learning (DL) from the cloud to the edge introduces a critical systems challenge: enabling efficient and stable distributed training under dynamic and heterogeneous conditions. Pipeline parallelism is crucial for training throughput, but existing frameworks, designed for cloud servers, incur high overhead and adapt poorly to dynamic, heterogeneous edges. To bridge this gap, we propose Chameleon, a synchronous and adaptive bidirectional pipeline framework that achieves high throughput in dynamic and heterogeneous edge systems. At its core, Chameleon incorporates a hardware-aware throughput model to capture pipeline stage performance on heterogeneous devices. This model enables a stage-adaptive runtime mechanism, featuring a scheduler that mitigates contention through dynamic task scheduling and a micro-batch optimizer that tunes per-stage batch sizes using real-time throughput feedback. Extensive experiments show that Chameleon outperforms state-of-the-art methods, improving training throughput by up to 2.1 × in dynamic and heterogeneous edge systems.
Tao Liu, Xiang-Peng Hou, Li-Ying Li et al.· Proceedings of the Internati...· 0 citations
Distributed AI training involves recurring rounds of data exchange between multiple pairs of GPU nodes. Slowdown in even one flow due to congestion can cause the entire communication round to slowdown. Current approaches for evading congestion in AI clusters assume global control over the entire workload (e.g. coordinating the schedule of all jobs) or assume infrastructural support (e.g. adaptive routing in switches). They are thus ill-suited in a shared cloud setting where AI jobs belonging to one user can face external congestion from other users'jobs or background traffic beyond its own control. In this paper, we build a system, REACT, that tunes the recurring pattern of data exchange between GPU nodes (known as communication collectives) in response to congestion. REACT works at the application (communication library) layer, where it detects congestion at runtime using readily available flow stats, and tunes the collective pattern to alleviate congestion - changing the set of incident flows while retaining the semantics of information exchange (e.g. selecting which node aggregates data in an AllReduce tree). REACT requires no explicit support from the underlying network infrastructure and can be unilaterally deployed by individual users in a shared cloud setting. We prototype REACT as a shim layer over NCCL, and evaluate it on a shared academic GPU cluster - enabling REACT improves communication performance (algorithm bandwidth) by 13%-38% under network congestion. Our simulations across a range of congestion scenarios further reveal up to 75% performance improvement, highlighting the effectiveness of our approach.
Eashan Gupta, Yong-Zhou Chen, Apoorve Mohan et al.· 0 citations
Large language models are increasingly trained on GPUs distributed across multiple regions, but geo-distributed training is challenging in practice. Real clusters often contain GPUs with different speeds and memory capacities, and they communicate over slow wide-area networks. Our analysis shows that this creates serious problems: existing synchronous methods preserve stable updates, but fast GPUs wait up to 20.9% of their runtime for slower ones, and all workers spend, on average, 65.8% of their runtime on synchronization. Recent asynchronous methods reduce waiting time but worsen the model accuracy due to stale updates. To address the problems, we present GeoMesh, a synchronous geo-distributed training framework for heterogeneous GPUs. GeoMesh balances per-worker workloads by assigning each GPU a suitable batch size and number of inner steps, so faster GPUs do more useful work instead of waiting. It also reduces communication volume by nearly 32x by exchanging compressed sign-based pseudo-gradients with lightweight magnitude and token count. Across heterogeneous GPUs and Azure-derived WAN, GeoMesh reduces time-to-target perplexity by up to 70.2% over representative baselines and lowers straggler- and WAN-induced GPU idle by up to 8.0x and 5.6x, respectively, while preserving comparable zero-shot accuracy.
Changyong Shin, Jaerim Park, Mi-Gyung Kang et al.· 0 citations
Preliminary results from a geo-distributed LLM training prototype that treats networking constraints as first-order design concerns are presented, motivating adaptive networking support for synchronization, compression, placement, telemetry, and recovery in geo-distributed LLM training.
Ziyue Luo, Jiaxuan Cai, Cedric Le Denmat et al.· Conference on Applications,...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
