Richard O. Sinnott

Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines

Kafila is proposed, whose protocol assembles a ring from behind NATs, preferring direct paths and relaying where traversal fails, while its planner measures each device's memory bandwidth, capacity and reachability, divides the model exactly for a fixed ring order, and places the head together with that division rather...