Selective activation checkpointing (SAC) decides which activations to save for the backward pass and which to recompute: each choice is a point in the plane of peak memory and recomputation time, and the goal is that plane's Pareto frontier. Current practice falls short of this goal. Hand-tuned policies fix a single po...
Kai-Hui Ren, Yi-Fan Zhao, Ze-Jian Wang et al.· 0 citations
Fully Sharded Data Parallelism (FSDP2) maintains parameters and gradients as per-parameter tensors, whereas its collective path operates on flat, rank-major buffers. Converting between these layouts copies all-gather outputs into parameter storage and packs gradients into full-group fp32 buffers. Layout conversion alon...
Ze-Jian Wang, Yi-Fan Zhao, Kai-Hui Ren et al.· 0 citations
Full 3D attention is costly in video Diffusion Transformers. Block-sparse attention reduces this cost by computing only selected attention blocks. However, existing methods either overlook the potential to optimize sparsity allocation or rely on heuristic rules or local metrics to guide it. In this work, we observe tha...
Tian-Ze Xia, Xiao-Ling Zhou, Ru-Ya Lan et al.· 0 citations
DraftAttention2 is presented, a training-free framework that uses the low-resolution draft attention map to jointly select attention blocks and assign their numerical precision, and achieves a superior quality-efficiency trade-off over existing efficient video generation methods.
Rui Ding, Haopeng Li, Wei-Ze Ma et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.