Latency-Aware Edge LLM Inference via Model Preloading and Semantic Knowledge Reuse

Abstract

Deploying large language models (LLMs) at the network edge requires low-latency inference despite limited GPU memory and frequent model switching. Conventional caching strategies are ineffective because semantically similar queries may differ significantly in surface form. To address this issue, we propose a unified framework that integrates adaptive LLM preloading with semantic-aware knowledge reuse. Specifically, we model semantic knowledge reuse as a three-stage mechanism, namely complete hit, partial refinement, and missing hit, and characterize the latency of each stage through an embeddingbased retrieval framework. We further formulate two separable latency-reduction problems for threshold optimization and preloading optimization, which can be efficiently solved via onedimensional grid search and linear programming, respectively. Simulation results show that the proposed approach significantly reduces service latency and improves scalability compared with both direct inference and cache-only baselines under constrained GPU memory.