#edge computing
Open access
Adaptive Hybrid Quantization Framework for Real-Time Speculative Decoding in Edge-Cloud Systems
Sep 2026 · Zenodo (CERN European Organization for Nuclear Research)
Abstract
High-throughput inference for Large Language Models (LLMs) remains bottlenecked by memory bandwidth constraints and compute latency, particularly in hybrid edge-cloud topologies. This paper proposes an Adaptive Hybrid Quantization (AHQ) framework integrated with real-time speculative decoding. By dynamically allocating low-bit precision formats (INT4/FP8) to target speculative draft models while preserving high-precision FP16 evaluation on base models, the proposed architecture significantly reduces memory footprint without sacrificing contextual coherence. Experimental benchmarks demonstrate a 2.8x speedup in token generation and a 42% reduction in compute overhead compared to monolithic baseline execution.
