LST-Sim: An Efficient Simulation Platform for Large-Scale Model Training
TL;DR
LST-Sim is presented, an efficient simulator designed for large-scale model training, balancing simulation accuracy and runtime efficiency, and a domain-based traffic management mechanism that integrates theoretical analysis with network simulation to handle different types of communication traffic, thereby improving simulation efficiency.
Abstract
Existing model training simulators fail to support heterogeneous GPU architectures and various intra-server high-speed interconnect protocols (e.g., NVLink, HCCS), while suffering from low efficiency in full-packet-level network simulation. In this paper, we present LST-Sim, an efficient simulator designed for large-scale model training, balancing simulation accuracy and runtime efficiency. First, the simulator takes into account the impacts of computational heterogeneity across different GPU models and inter-GPU interconnection bandwidth on training latency, and constructs a theoretical model. Furthermore, we propose a domain-based traffic management mechanism that integrates theoretical analysis with network simulation to handle different types of communication traffic, thereby improving simulation efficiency. The effectiveness of LST-Sim is well validated on the Ascend 910B training cluster. Its simulation results are in close agreement with real-world measurements. Meanwhile, the domain-based traffic management mechanism significantly reduces simulation runtime by approximately 75% while maintaining high fidelity.