Cambricon-FlexLLM: A Flexible Chiplet-Based Hybrid Architecture for On-Device 70B LLM Inference

Abstract

Deploying advanced large language models on edge devices, such as smartphones and robotics, is a growing trend that enhances user data privacy and network connectivity resilience while preserving intelligent capabilities. However, such a task exhibits single-batch computing with incredibly low arithmetic intensity, which poses significant challenges of huge memory footprint and bandwidth demands on limited edge resources. To address these issues, we introduce Cambricon-FlexLLM, a chiplet-based hybrid architecture with an NPU and a dedicated NAND flash chip to enable efficient on-device inference of 70B LLMs. Such a hybrid architecture utilizes both the high computing capability of NPU and the data capacity of the NAND flash chip, with the proposed hardware-tiling strategy that minimizes the data movement overhead between NPU and NAND flash chip. Specifically, the NAND flash chip, enhanced by our innovative in-flash computing and on-die ECC techniques, excels at performing precise lightweight on-die processing. Simultaneously, the NPU collaborates with the flash chip for matrix operations and handles special function computations beyond the flash’s on-die processing capabilities. Furthermore, to exploit the activation sparsity prevalent in modern LLMs, we propose a co‑activation neuron‑inspired weight‑reordering algorithm and a sparsity‑aware dynamic partitioning scheme. These optimizations significantly reduce the transfer of ineffective weight data. Experimental results demonstrate that Cambricon-FlexLLM achieves an inference speed of 3.44 tokens/s for 70B LLMs and 36.34 tokens/s for 7B LLMs, outperforming state‑of‑the‑art flash‑offloading frameworks by 22 × –45 ×. Leveraging activation sparsity, Cambricon-FlexLLM achieves an additional 1.7× average speedup, ranging from 1.3× to 2.0× compared to dense inference. These results show a path toward local 70B-class decode for privacy-sensitive, offline, and fallback use cases on resource-constrained edge devices, while also highlighting the need to manage prefill latency, energy, thermal behavior, and shared-storage constraints.