DABO: Difficulty-Aware Binary Offloading for Collaborative Large-Small Model Inference

Abstract

Large-small model collaboration, as an emerging paradigm for efficient language model inference, plays a significant role in balancing inference quality and service efficiency in cloud-edge intelligence systems. By combining the strong generalization capability of cloud-side large language models (LLMs) with the low-latency inference capability of edge-side small language models (SLMs), this paradigm provides a practical way to serve heterogeneous language requests. However, existing collaborative inference methods still face a critical challenge: forwarding all requests to the cloud leads to high latency and unnecessary service cost, while relying only on edge-side SLMs may degrade prediction quality for difficult inputs. Therefore, we propose DABO, a calibration-aware binary offloading method for collaborative large–small model inference. Specifically, the edge-side SLM first performs lightweight probe inference and produces a prediction confidence distribution. DABO then calibrates the confidence through temperature scaling and converts it into a label-space-normalized difficulty score. Based on a model–task-specific operating threshold selected under an explicit accuracy constraint, reliable requests are answered locally by the edge-side SLM, whereas uncertain requests are selectively escalated to the cloud-side LLM. Extensive experiments demonstrate that DABO maintains competitive end-to-end accuracy while processing an average of 83.72% of requests at the edge, reducing expected calibration error by 31.08%, and decreasing mean latency by 84.24% compared with Cloud-only inference in a controlled workstation-based edge simulation.