Efficient VLM Inference System With LoRA Adapters at the Edge

Abstract

Vision Language Models (VLMs) extend large language models with visual perception, enabling complex vision tasks at the edge. Low-rank adaptation (LoRA) adapters offer a lightweight method to inject domain-specific knowledge into a shared base VLM, making them attractive for edge serving where concurrent mobile clients require diverse specialized capabilities. However, existing LoRA serving systems, designed primarily for text workloads, fail to meet the efficiency and accuracy demands of vision applications: heterogeneous adapter batching incurs excessive padding waste, mode-switch mechanisms clash with modern serving techniques causing request starvation, and no principled method exists for generating accuracy-guaranteed adapters. We present VaLoRA, an end-to-end LoRA VLM serving system that addresses these challenges with three techniques. First, an accuracy-aware adapter generator formulates adapter creation as constrained bin packing and pairs it with a hierarchical runtime router for automatic request dispatch. Second, an Adaptive-Tiling Matrix Multiplication (ATMM) operator eliminates padding waste by selecting optimal tiling configurations for concurrent heterogeneous adapters. Third, a flexible orchestrator enables swift mode switching and mixture inference to satisfy diverse latency and throughput requirements without request starvation. Evaluation on five vision tasks across three VLMs shows that VaLoRA improves accuracy by 24–62% over original VLMs and reduces latency by 20–89% compared to S-LoRA, Punica, and dLoRA.