A Survey on Optimization Techniques for On-Device Large Language Model Inference

Abstract

Large Language Models (LLMs) have brought transformative advancements to natural language processing. However, their massive computational and memory requirements—stemming from tens of billions of parameters—make deployment on resource-constrained edge devices such as mobile phones and embedded systems highly challenging. Consequently, extensive research efforts have focused on enabling efficient on-device LLM inference. This paper provides an in-depth survey of two major inference acceleration techniques for on-device LLMs: KV cache optimization and speculative decoding. We present the core principles, representative studies, and performance characteristics of each approach. Furthermore, we discuss open challenges and future research directions, aiming to contribute to the advancement of efficient on-device LLM systems.