Scaling LLMs via Collaboration: Efficient LLM Inference Across Resource-Constrained Devices

Abstract

Large language models (LLMs) deliver remarkable capabilities but remain difficult to deploy on memory- and compute-limited edge devices. A natural question is whether multiple devices can collaboratively execute token-by-token inference with low latency. Existing parallelism schemes, designed for datacenter GPUs and batch-oriented workloads, break down on edge devices due to ill-suited partitioning and heavy synchronization overheads. We identify three core challenges: (i) per-token synchronization dominates latency in decoding, (ii) naive scaling amplifies communication cost, and (iii) Transformer operators harbor hidden heterogeneous computation–communication characteristics that remain underutilized. Our key insight is that partitioning and synchronization must be co-optimized to unlock scalable edge collaboration. We present CoSEI, a system that introduces operator-aware partitioning and synchronization-minimizing execution, guided by a lightweight planner and a tailored runtime. A prototype built atop llama.cpp achieves up to 5.4× lower latency than pipeline baselines on an eight-node edge device cluster, while enabling models that exceed single-device memory. Results demonstrate that with the right design, collaborative edge inference can make LLMs practical beyond the cloud.