Communication-Efficient Decentralized LLM Inference over Low-Bandwidth Distributed Nodes
TL;DR
BandwidthLLM, a communication-efficient framework that integrates three techniques: a bandwidth-aware layer placement algorithm that minimizes boundary-level transfer cost according to link bandwidth and node reliability; a lightweight activation compression scheme combining adaptive quantization with outlier-aware clipping and error feedback; and a semantic preservation check that automatically falls back to higher precision when compressed activations deviate beyond a calibrated threshold.
Abstract
Decentralized large language model (LLM) inference distributes transformer layers across heterogeneous nodes to enable resource-constrained participants to collaboratively serve large models. However, inter-node activation transfer constitutes a significant communication bottleneck, especially under low-bandwidth conditions prevalent in consumer-grade and edge networks. We propose BandwidthLLM, a communication-efficient framework that integrates three techniques: (1) a bandwidth-aware layer placement algorithm that minimizes boundary-level transfer cost according to link bandwidth and node reliability; (2) a lightweight activation compression scheme combining adaptive quantization with outlier-aware clipping and error feedback; and (3) a semantic preservation check that automatically falls back to higher precision when compressed activations deviate beyond a calibrated threshold. Experiments on Llama-2-7B, Mistral-7B, and Qwen2.5-7B across simulated bandwidth conditions (20–500 Mbps) demonstrate that BandwidthLLM reduces inter-node activation traffic by 62.7–63.4% relative to the BloomBee-like baseline while maintaining perplexity degradation of at most 0.31 points and task accuracy within 0.9 percentage points of uncompressed inference. Under 20 Mbps links, BandwidthLLM achieves 1.7× higher throughput and 41% lower latency.