Jul 2026· International Conference on Ubiquitous and Future Networks· pp. 765-767· 0 citations· 23 references
Abstract
Large Language Models (LLMs) have brought transformative advancements to natural language processing. However, their massive computational and memory requirements—stemming from tens of billions of parameters—make deployment on resource-constrained edge devices such as mobile phones and embedded systems highly challenging. Consequently, extensive research efforts have focused on enabling efficient on-device LLM inference. This paper provides an in-depth survey of two major inference acceleration techniques for on-device LLMs: KV cache optimization and speculative decoding. We present the core principles, representative studies, and performance characteristics of each approach. Furthermore, we discuss open challenges and future research directions, aiming to contribute to the advancement of efficient on-device LLM systems.
Results show that most models above 500 million parameters were not suitable for the target environment, while a subset of 4-bit quantized models achieved stable execution and acceptable response quality, and provide practical guidance for integrating generative AI into low-resource consumer hardware.
Suayder M. Costa, Igor Lima, William Harada et al.· Anais do XVIII Simpósio Bras...· 0 citations
Evaluation across multiple datasets shows that SelectInfer achieves significant reductions in memory footprint and computation while preserving task performance, making it a practical step towards enabling LLM deployment on edge devices.
Huzaifa Shaaban Kabakibo, Eric Schniedermeyer, Artem Burchanow et al.· 0 citations
This study presents the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks and three hardware backends, and identifies a distinct phase split where NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decod...
Guanyu Cai, Ruiming Tian, Lang Yang et al.· 0 citations
These results show that selecting lightweight LLMs by size, FLOPs, latency, or accuracy alone can select the wrong deployment candidate; PTME profiling exposes configurations that preserve useful accuracy at lower physical cost.
Tomohiro Harada, Enrique Alba, Gabriel Luque· arXiv.org· 0 citations
Five self-speculative decoding techniques are characterized across three model sizes and three datasets and recommendations for future research in this area are provided.
Jungmin Ha, Karthik Ganesan, Anh Nguyen et al.· 0 citations
An MLIR (Multi-Level Intermediate Representation) based compilation method for large language models, illustrated using two dialects of operators, TopOp and TpuOp, supporting a variety of generative models including the Qwen, Llama, InternVL, and MiniCPM-V series.
Pengchao Hu, Zhibin Xin, Yifan Chen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
