This work proposes PrefillShare, a novel algorithm that enables sharing the prefill stage across multiple fine-tuned models in a disaggregated setting and introduces a routing mechanism that enables effective prefill sharing in a vLLM-based disaggregated system.
Sunghyeon Woo, Hoseung Kim, S. Shim et al.· arXiv.org· 3 citations
SlimWise is presented, a serving framework that tailors the expert pool to each inference phase that improves decode throughput by up to 1.81x at 50% expert pruning with minimal accuracy loss and shows that benchmark accuracy can conceal substantial pruning-induced changes in generation length.
Gunho Park, Kyoungho Jeun, Juntaek Oh et al.· arXiv.org· 1 citation
Block Verification-aware loss (BV loss), a training objective designed to maximize the expected acceptance length of a drafted sequence, outperforms tokenwise acceptance objectives such as TV loss and LK loss, and its gains extend to token verification and greedy decoding.
Suyoung Kim, Jahyun Koo, Hyeonjin Kim et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.