A3D-MoE: Acceleration of Large Language Models With Mixture of Experts via 3-D Heterogeneous Integration
TL;DR
A3D-MoE addresses large language models' challenges with 3-D heterogeneous integration to improve memory bandwidth and reduce NoC overhead/energy, and a hardware resource-aware operation fusion scheduler that fuses attention/MoE operations to boost performance.
Abstract
Conventional large language models (LLMs) have large parameter sets, making inference costly and energy-intensive; mixture-of-experts (MoEs) mitigates this by activating fewer weights per token, but fine-grained MoE LLMs still face runtime workload variability, inefficient conventional scheduling, and high bandwidth memory (HBM) loading energy/bandwidth demands. A3D-MoE addresses these with 3-D heterogeneous integration to improve memory bandwidth and reduce NoC overhead/energy; a 3-D-systolic array with V-Cache reuse and unified 3-D dataflow to optimize hardware utilization across workloads; a hardware resource-aware operation fusion scheduler (HR-OFS) that fuses attention/MoE operations to boost performance; and MoE Score-Aware HBM access reduction with even–odd expert placement (MoE-HBMR-EOP) to reduce HBM access/bandwidth demand. Evaluations show <inline-formula> <tex-math notation="LaTeX">$1.8\times $ </tex-math></inline-formula>–<inline-formula> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula> lower latency, <inline-formula> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula>–<inline-formula> <tex-math notation="LaTeX">$4\times $ </tex-math></inline-formula> lower energy, and <inline-formula> <tex-math notation="LaTeX">$1.44\times $ </tex-math></inline-formula>–<inline-formula> <tex-math notation="LaTeX">$1.8\times $ </tex-math></inline-formula> higher throughput than the state-of-the-art.