DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory
Existing LLM serving systems virtualize and optimize KV-cache memory, but treat model-weight memory as fixed throughout execution. Recent work on multi-precision model representations challenges this design by allowing a single stored model to support both full-accuracy and lower-precision execution, making the effecti...
Xuan Truong Nguyen, Tien-Son Pham, Tuan-Duc Chu et al.
· 0 citations