Spurthi Tallam

Twice as Fast, Until It Isn't: Quantized Serving of Mid-Size Open LLMs

A team serving a 9B-20B open-weight language model on vLLM or SGLang must choose a quantization recipe, and needs two numbers: the speedup over BF16 and the accuracy given up. We assemble every public measurement of these numbers that we could verify and find that the record supplies them only in pieces. Only two check...