Agentic LLM workflows issue many dependent calls with unpredictable resource demand, causing queue buildup and latency degradation on shared serving backends when left unmanaged. In this paper, we propose CALM-MAS, a congestion-aware serving framework for LLM applications that treats LLM test-time computation as an elastic resource, dynamically adjusting the compute profile of admitted tasks to tame congestion. CALM-MAS detects early signals of back-end saturation, and leverages the flexibility of LLM applications to regulate load. During spikes of requests, the system downgrades agent topology and reasoning depth; during low-utilization periods, it allocates additional reasoning effort to maximize task accuracy. Compared with a static serving baseline based on vLLM, CALM-MAS reduces shared-backend tail latency by 77% with the accuracy degradation remaining confined to 6.1 pps relative to the native agent configuration.
Mouheb Ben Nasr, Muhammad Bilal, Alessandro Cornacchia et al.· Proceedings of the 17th ACM...· 0 citations
Embodied LLM systems increasingly co-locate latency-critical robotics pipelines with compute- and memory-intensive language-model inference on edge platforms such as NVIDIA Jetson. This co-location avoids cloud round trips and enables privacy-preserving, low-latency interaction, but it also creates a new source of resource interference. Our experiments show that the deadline miss rate of a real-time task can reach nearly 100% under co-running LLM inference. The root cause is that, although robotics workloads and LLM inference typically execute on the CPU and GPU, respectively, they share the same unified-memory subsystem on edge platforms. As a result, Memory-Controller (MC) capacity can become a hidden bottleneck even when conventional compute resources do not appear saturated. Based on this finding, we propose MCSched, a lightweight MC-aware runtime scheduler for embodied LLM systems. MCSched combines coarse MC-pressure signals with foreground timing feedback and temporarily gates background LLM execution when robotics tasks approach deadline risk. In our prototype experiments, MCSched successfully reduced the deadline miss rate of the camera task and PCL task by up to 83.35% and 74.11%, respectively, on the ROS-LLM framework while maintaining LLM inference speed. These observations provide preliminary but compelling evidence that MC-aware scheduling is a practical operating system/runtime direction for deployable embodied AI on unified memory edge devices.
Teng Mei, Cheng-Xuan Pei, Marco Canini et al.· Proceedings of the 17th ACM...· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
