Vikram Sharma Mailthody

Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamo

Large language model (LLM) inference replicas run across tightly coupled GPUs and serve traffic continuously for weeks. Hardware and software failures are therefore inevitable, and one worker failure can disrupt an entire replica. Recovery requires reinitializing the engine, taking minutes even when weights and compila...