Batch processing is useful for optimizing the utilization of various hardware resources, such as GPUs, CPUs, and other specialized accelerators, and is also important for improving the throughput of LLM serving. Many research works have been conducted for better batching and scheduling policies for the efficiency of cl...
Jin-Xin Yu, Lei Wang, Meng-Di Wang et al.· IEEE Transactions on Paralle...· 0 citations
The rapid adoption of LLM applications is driving up serving cost, yet GPUs remain underutilized in practice due to single-model provisioning and highly variable, long-tailed workloads. Co-locating two or more LLMs on one GPU can improve utilization, but it requires fast and coordinated control of both KV cache/paramet...
Cun-Chi Lv, Xiao Shi, Zheng-Yu Lei et al.· IEEE International Conferenc...· 0 citations
The growing demand for long-context LLM inference has exposed a critical bandwidth–capacity trade-off in memory systems, rendering single-tier PIM architectures ineffective. HBM-PIMs offer high bandwidth but limited capacity, while DIMM-PIMs provide scalability at the cost of lower bandwidth; neither satisfies the thro...
Shixin Zhao, Lian Liu, Xiangwen An et al.· IEEE transactions on compute...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.