Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines
Kafila is proposed, whose protocol assembles a ring from behind NATs, preferring direct paths and relaying where traversal fails, while its planner measures each device's memory bandwidth, capacity and reachability, divides the model exactly for a fixed ring order, and places the head together with that division rather...
Murtaza Rangwala, Richard O. Sinnott, Rajkummar Buyya
· 0 citations