Batch processing is useful for optimizing the utilization of various hardware resources, such as GPUs, CPUs, and other specialized accelerators, and is also important for improving the throughput of LLM serving. Many research works have been conducted for better batching and scheduling policies for the efficiency of cl...
Jin-Xin Yu, Lei Wang, Meng-Di Wang et al.· IEEE Transactions on Paralle...· 0 citations
Mixture-of-Experts (MoE) have become a mainstream way to scale Large Language Models (LLMs), because each token activates only a small subset of experts, reducing inference cost. However, MoE decode remains highly memory-bound and exhibits strongly skewed, time-varying expert loads. 3D-stacked DRAM near-memory processi...
Yu-Dong Pan, Lian Liu, Shi-Xin Zhao et al.· 0 citations
The growing demand for long-context LLM inference has exposed a critical bandwidth–capacity trade-off in memory systems, rendering single-tier PIM architectures ineffective. HBM-PIMs offer high bandwidth but limited capacity, while DIMM-PIMs provide scalability at the cost of lower bandwidth; neither satisfies the thro...
Shixin Zhao, Lian Liu, Xiangwen An et al.· IEEE transactions on compute...· 0 citations
PFM (PIM-as-Flexible-Memory), a dual-view memory system that decouples physical data layout from accessor-visible logical views, is presented, demonstrating its effectiveness and broad applicability as a unified memory management solution for NPU-PIM systems.
Shixin Zhao, Lian Liu, Tian Han et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
