Unleashing Chain-of-Thought at the Network Edge: A Robust Microservice Framework

Abstract

Foundation models (FMs) utilizing Chain-of-Thought (CoT) reasoning are pivotal for next-generation Edge Intelligence but incur substantial computational overheads. To address the latency and privacy limitations of centralized cloud inference, collaborative edge inference offers a promising solution by virtualizing reasoning steps into distinct microservices across heterogeneous devices. However, the generative uncertainty of CoT and the volatility of wireless channels pose significant challenges to system reliability and efficiency. In this paper, we propose a robust microservice orchestration and radio resource allocation framework for edge CoT reasoning. To mitigate the non-stationarity of workloads during inference, a token-oriented Lyapunov optimization framework is developed to track sequential energy consumption under dynamic CoT workloads. Furthermore, motivated by an R-contamination uncertainty model, we formulate a robust decision model for jointly balancing latency and energy consumption. To obtain an implementable scheduler without explicit transition kernels, we develop a robust SAC-based method with an LSE-conservative critic target and establish a finite-time stationarity result for the implemented smoothed surrogate under stated assumptions. Simulation results show that the proposed scheduler attains the target average J/token in the main evaluation and achieves a favorable latency–energy trade-off against the considered benchmarks.