Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

Master Computer Science

Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels

Edge LLM deployment is shaped by more than model size and precision: inference backend, hardware platform, memory traffic, and power management all affect latency and efficiency. We present Hydra, a common-schema, phase-aware workload characterization framework for LLM inference on edge SoCs. Hydra instruments HuggingFace Transformers and llama.cpp with a shared per-prompt timing schema and fuses those records with hardware telemetry, enabling a multi-dimensional characterization of performance, system-resource utilization, and efficiency across prefill and decode phases. Using Hydra, we evaluate three consecutive edge System-on-Chip (SoC) generations (AGX Xavier, AGX Orin, and AGX Thor), 13 instruction-tuned LLMs from seven families, five execution formats, and consider input/output-length sensitivity. The resulting artifact contains roughly 107K per-prompt records and is publicly released with Hydra. Our analysis shows that aggregate latency alone hides key deployment effects: backend structure changes where latency is introduced, quantization reduces memory traffic and energy but does not predict power monotonically, and SoC generation changes how utilization and efficiency should be interpreted. By connecting phase-level timing with system-resource utilization and efficiency metrics, Hydra enables reproducible, phase-aware characterization of edge LLM inference. Hydra's source code and the collected per-prompt trace corpus are available open-source at: https://github.com/amirtaherin/hydra

Characterizing self-speculative decoding approaches for accelerating LLMs

Evaluating large language model compression: a comparative analysis on state-of-the-art models across diverse hardware platforms

This work presents a systematic, empirical comparison of contemporary compression techniques for large language models (LLMs), namely quantization, pruning, and parameter-efficient fine-tuning (PEFT) using a representative set of open-source model families (Llama, Mistral, Phi and Qwen) and model scales (1.7 Billion to 70 Billion). Evaluation combined benchmarks (MMLU, SQuAD v2, TinyBenchmarks and WikiText), deployment metrics (peak memory, time-to-first-token, tokens/sec and maximum sequence lengths) and settings (multi-GPU clusters, single-GPU PC, laptop, and smartphone) to capture real-world trade-offs. Quantization often delivered the best wins for deployment feasibility—enabling single-device and mobile inference—but required careful per-model tuning and backend support to avoid throughput regressions. Pruning reduced parameter counts substantially but frequently incured large, even catastrophic, performance loss beyond moderate sparsity levels. Retraining partially mitigated this but did not uniformly close the gap to quantization. Finally, PEFT methods enabled models to match or outperform models with up to 18 times the parameters on SQuAD v2 while reducing storage as well as optimizer overhead and often improved task performance even when full fine-tuning failed.

Rethinking LLM Deployment for Intent-Based Serving

Large Language Models (LLMs) expose a large deployment configuration space spanning parallelism and compression techniques, with each configuration introducing different tradeoffs in latency, memory consumption, cost, and output quality. Existing systems either rely on expensive profiling across deployment configurations or inefficiently utilize fragmented GPU resources in multi-tenant clusters. We present MaverIQ, an intent-based LLM inference serving system that automatically maps user intents to deployment configurations while minimizing operational cost for the provider. To reduce profiling overheads, MaverIQ introduces lightweight LLM fingerprints and analytical models that extrapolate latency and memory footprint from only a few observations. To efficiently utilize fragmented GPU resources, MaverIQ leverages our observation that, unlike training, unevenly distributing LLM layers across GPUs has little impact on inference latency. Our evaluation shows that MaverIQ reduces profiling cost by 7-15× compared to state-of-the-art baselines and reduces operational cost by 3.8-8.3× across diverse LLMs, traces, and loads while effectively meeting user intents. Our code is available at https://github.com/UT-SysML/MaverIQ.

S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices

Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency, naively combining them often incurs excessive verification overhead and poor expert reuse, limiting their effectiveness in memory-bound edge settings. In this work, we propose S2-MoE, an efficient self-speculative decoding framework for MoE inference on edge devices. S2-MoE reduces redundant verification through routing-aware adaptive speculative expansion, improves verification efficiency with reuse-aware expert gating, and aligns draft and target execution via shared context. Implemented in llama$.$cpp, S2-MoE achieves up to $5.3\times$ speedup (about $2.0\times$ on average) over standard autoregressive decoding across diverse MoE models and datasets on edge devices. Code is available at https://github.com/angerybob/S2-MoE.