The Mixture-of-Experts (MoE) architecture has become the dominant paradigm for scaling Large Language Models, yet its explosive parameter growth routinely overflows the GPU memory budget. Weight quantization offers a practical way to fit these models into limited GPU memory, but reducing precision below 4 bits causes severe accuracy loss. Existing approaches, mixed-precision quantization and low-rank compensation, have been treated as independent strategies, overlooking the fact that both draw from the same hardware memory budget. Consequently, optimizing either dimension in isolation systematically overshoots on some experts while starving others, leaving the achievable memory–accuracy frontier well below what a joint allocation could reach. We present BR-MoE, a memory-budget-aware co-design framework that treats per-expert bit-width and compensator rank as joint variables under a single memory constraint. Guided by a lightweight layer-wise sensitivity proxy, BR-MoE casts the global allocation task as a Multiple-Choice Knapsack Problem (MCKP) and solves it to optimality via Integer Linear Programming (ILP). To turn memory savings into real inference speedups, BR-MoE further integrates a fused grouped expert GEMM backend that consolidates fragmented per-expert computations into bandwidth-efficient kernels. Evaluated across multiple MoE models on NVIDIA A100 GPUs, BR-MoE establishes a superior memory–accuracy trade-off over state-of-the-art quantization methods, delivers substantial end-to-end inference speedups, and re-derives an optimal allocation for any new memory budget in seconds, making it a deploy-time tunable system rather than a one-shot compression recipe.
Lei Jiang, Jingwei Sun, Jun-Qing Lin et al.· Proceedings of the Internati...· 0 citations
Large Language Models (LLMs) have demonstrated remarkable capabilities, yet their extensive parameter scales pose significant challenges for practical deployment. Unstructured pruning has emerged as an effective model compression strategy with minimal performance loss, which introduces fine-grained sparsity for weight parameters. While existing methods employ a layer-wise pruning strategy to avoid the complexity of global pruning for billion-scale LLMs, they require appropriate sparsity allocation for the layer-wise pruning objectives and often lead to suboptimal solutions for the overall model. In this paper, we propose Lua-LLM ( L earning u nstructured-sparsity a llocation in LLMs), a learning-based global pruning framework that explores the optimal unstructured sparsity allocation. Unlike existing pruning methods, which primarily focus on allocating per-layer sparsity, Lua-LLM achieves flexible allocation for both layer-wise and intra-layer sparsity. Furthermore, Lua-LLM leverages a soft Top-K operator to approximate the importance-based mask selection mechanism, enabling efficient binary mask learning. Experimental results on LLaMA and OPT families demonstrate significant performance improvements over existing methods.
Ming Lu, Jingwei Sun, Junqing Lin et al.· Neural Information Processin...· 1 citation
FlashQuant fuses the dense GEMM and sparse outlier SpMM paths into a single GPU kernel, enabling on-chip reuse of activation and output tiles across heterogeneous computations, and introduces three key techniques: sparse-dense tiling, which aligns outlier processing with dense GEMM tiles, and pipelined scheduling, which overlaps computation with data movement.
Jun-Qing Lin, Jingwei Sun, Zhengding Hu et al.· 0 citations
DeltaLog is presented, a recurrent-state decoding scheme that reduces this overhead without changing the model semantics, and is implemented for GDN, KDA, and RWKV6 and integrated into a prototype serving stack.
Hardware-agnostic strategies for accelerating text-to-image diffusion, such as timestep distillation and feature caching, can reduce inference time without custom kernels or system-level optimization. Among them, multi-resolution generation strategies have recently received broad attention, attaining more than 5x speedup without any training. However, the design of performing upsampling in the latent space, together with the selective modification of partial regions, causes these methods to exhibit noticeable blurring or artifacts. To this end, we propose MrFlow, a training-free multi-resolution acceleration strategy for pretrained flow-matching models built upon a staged low-to-high-resolution pipeline. MrFlow first rapidly generates the main structure at low resolution, then performs super-resolution in the pixel space using a lightweight pretrained GAN-based model, subsequently injects low-strength noise to enable high-frequency resampling, and finally refines the details at high resolution. Quantitative and qualitative results on FLUX.1-dev and Qwen-Image show that MrFlow exploits the quadratic token reduction and reduced step requirement of low-resolution sampling to achieve 10x end-to-end acceleration while keeping OneIG within a 1% gap relative to that before acceleration, significantly surpassing other training-free acceleration strategies, and requiring no training or runtime dynamic identification whatsoever. MrFlow can further be directly combined orthogonally with pre-trained timestep distillation strategies, achieving even higher generation acceleration of up to 25x.
Xingyu Zheng, Xianglong Liu, Yifu Ding et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
