TL;DR
A time-varying integer program to minimize the long-term total cost of the edge AI inference system, including the inference latency, the inference error rate, the query-dispatching communication cost, and the energy consumption, subject to resource and workload constraints is proposed.
FAST is presented, a holistic framework that accelerates end-to-end TGNN training by jointly optimizing sampling, memory I/O, and computation and designs thread-efficient graph operators tailored to sparse temporal subgraphs, improving GPU cache locality and reducing the latency of aggregation and edge softmax.
With the growing deployment of Deep Neural Networks (DNNs) in latency-critical services, optimizing inference efficiency on GPUs has become crucial. While exploiting operator parallelism offers a promising avenue to accelerate inference and improve hardware utilization, existing approaches often overlook two critical f...
This work introduces a tile-aware scheduling framework for efficient sparse Vision Transformer execution on GPUs and introduces a training-aware extension that reuses the inference tile schedule and augments it with backward computation and activation-memory strategies.
This work highlights the need for NUNA-aware routing (NAR), choosing optimized, spatially-aware inter-GPU paths in large scale-up network topologies and introduces NUNA-aware placement (NAP), placing threadblocks and data near I/O to optimize the inter-GPU traffic.
ElastiCo is presented, an elastic co-location framework that enables training and inference workloads to safely share GPUs through three integrated mechanisms that decomposes the resulting multi-resource allocation problem into per-job configuration selection subproblems via dynamic per-resource shadow prices.
With the rapid advancement of deep learning technology, the parameter scale of large language models has grown exponentially, expanding from hundreds of millions in the early stages to hundreds of billions or even trillions today. Pipeline inference is a crucial approach enabling efficient inference in large language m...
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
