Large transformer-based models increasingly depend on multi-GPU execution, which requires frequent collective communication among GPUs. Existing communication libraries often rely on many GPU threads to achieve high bandwidth or low latency, resulting in a large streaming multiprocessor (SM)-side resource footprint. Th...
Keyvan Dadashzadeh, Yue-Hong Zhou, Min-Yu Cui et al.· 0 citations
A wave-quantized analytical model and launch-time resource manager for dependency-coupled overlap pipelines that selects the communication-CTA count and resource partition before each launch without candidate execution, per-workload profiling, or kernel recompilation is presented.
An automated framework is introduced that jointly explores compiler flags and runtime parameters using both Bayesian optimization and a tabu-search–based strategy to investigate the transferability of compile-time and runtime autotuning decisions across CPUs and GPUs, focusing on AdaptiveCpp, a SYCL implementation buil...
Hari Abram, Nikela Papadopoulou, Jens Domke et al.· International Conference on...· 0 citations
DEFT is presented, an energy-aware scheduling framework that jointly optimizes task-to-device assignment and per-GPU DVFS configuration for task-based multi-GPU applications and employs a cost-model–driven strategy that integrates slack awareness, throughput awareness, and explicit modeling of task execution cost, inte...
Jing Chen, Miquel Pericàs· International Conference on...· 0 citations
This paper presents FlashAttention-V, a blocked FlashAttention for scalable vector architectures that adapts efficiently from short to very long vectors by exploiting parallelism across attention heads, inter-head packing to enable efficient utilization of vector lengths beyond the head dimension, and improving vector...
