Pavle Glušac

Introducing Cluster Efficiency to LLM Serving: Pooling Economical Accelerators over Fast Interconnects

Long-context large language model (LLM) serving places substantial demands on compute, memory capacity, and memory bandwidth. Economical accelerators can offer these resources at lower cost, but their limited per-device capacity requires pooling across devices, making communication overhead critical. We study cluster e...