Skip to content
Log in
Sponsored Artificial Intelligence: A Modern Approach, Global Edition Artificial Intelligence: A Modern Approach, Global Edition

Author

Ming Lu

View on Semantic Scholar

2 papers indexed here

We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.

Not the right person? Other researchers publish under this name.

#large language models Book Open access Sep 2026

BR-MoE: A Memory-Budget-Aware Co-Design Framework for Efficient MoE Inference on Resource-Constrained GPUs

The Mixture-of-Experts (MoE) architecture has become the dominant paradigm for scaling Large Language Models, yet its explosive parameter growth routinely overflows the GPU memory budget. Weight quantization offers a practical way to fit these models into limited GPU memory, but reducing precision below 4 bits causes severe accuracy loss. Existing approaches, mixed-precision quantization and low-rank compensation, have been treated as independent strategies, overlooking the fact that both draw from the same hardware memory budget. Consequently, optimizing either dimension in isolation systematically overshoots on some experts while starving others, leaving the achievable memory–accuracy frontier well below what a joint allocation could reach. We present BR-MoE, a memory-budget-aware co-design framework that treats per-expert bit-width and compensator rank as joint variables under a single memory constraint. Guided by a lightweight layer-wise sensitivity proxy, BR-MoE casts the global allocation task as a Multiple-Choice Knapsack Problem (MCKP) and solves it to optimality via Integer Linear Programming (ILP). To turn memory savings into real inference speedups, BR-MoE further integrates a fused grouped expert GEMM backend that consolidates fragmented per-expert computations into bandwidth-efficient kernels. Evaluated across multiple MoE models on NVIDIA A100 GPUs, BR-MoE establishes a superior memory–accuracy trade-off over state-of-the-art quantization methods, delivers substantial end-to-end inference speedups, and re-derives an optimal allocation for any new memory budget in seconds, making it a deploy-time tunable system rather than a one-shot compression recipe.

Lei Jiang, Jingwei Sun, Jun-Qing Lin et al. · 0 citations
2025

Lua-LLM: Learning Unstructured-Sparsity Allocation for Large Language Models

Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their extensive parameter scales pose significant challenges for practical deployment. Unstructured pruning has emerged as an effective model compression strategy with minimal performance loss, which introduces fine-grained sparsity for weight parameters. While existing methods employ a layer-wise pruning strategy to avoid the complexity of global pruning for billion-scale LLMs, they require appropriate sparsity allocation for the layer-wise pruning objectives and often lead to suboptimal solutions for the overall model. In this paper, we propose Lua-LLM ( L earning u nstructured-sparsity a llocation in LLMs), a learning-based global pruning framework that explores the optimal unstructured sparsity allocation. Unlike existing pruning methods, which primarily focus on allocating per-layer sparsity, Lua-LLM achieves flexible allocation for both layer-wise and intra-layer sparsity. Furthermore, Lua-LLM leverages a soft Top-K operator to approximate the importance-based mask selection mechanism, enabling efficient binary mask learning. Experimental results on LLaMA and OPT families demonstrate significant performance improvements over existing methods.

Ming Lu, Jingwei Sun, Junqing Lin et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.