Jun Li

OmniPipe: Efficient, Flexible and Scalable Pipeline Parallelism for Large Model Training

OmniPipe is proposed, a flexible bidirectional multi-pipeline parallelism scheme for unified dense and MoE LLM training that minimizes the pipeline bubble ratio while effectively overlapping EP communication with computation, enabled by the flexible and scalable parallelism scheme of bidirectional pipelines.

OmniPipe: Efficient, Flexible and Scalable Pipeline Parallelism for Large Model Training

Mixture-of-Experts (MoE) has become the de facto architecture for scaling large language models, offering expanded capacity with manageable compute. Pipeline parallelism (PP) is indispensable for distributed MoE training, but state-of-the-art PP schemes face three major limitations: large pipeline bubbles, high per-sta...

Chat with the manufacturing process: A knowledge-guided cross-modal large language model for machining condition predictive maintenance