Jul 2026
NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement
NELSSA is presented, an LLM serving system that integrates GPUs with real-world Processing-near-Memory accelerator devices to efficiently support mixed-length workloads and improves decode throughput by up to 5.5x in tokens/sec and reduces P99 latency by up to 15x compared to GPU-only baselines.
Sookyung Choi, Seungyong Lee, Kangkyu Park et al.
· arXiv.org · 0 citations