Preprint
Sep 2026
Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamo
Large language model (LLM) inference replicas run across tightly coupled GPUs and serve traffic continuously for weeks. Hardware and software failures are therefore inevitable, and one worker failure can disrupt an entire replica. Recovery requires reinitializing the engine, taking minutes even when weights and compila...
Schwinn Saereesitthipitak, Mohammed H. Abdulwahhab, Hannah Zhang et al.
· 0 citations