Across autoregressive language-model pretraining, SOLAR improves final perplexity over tuned static schedules and automatic LR tuners for dense models from 60M to 1B, AdamW and Muon, and two MoE settings up to 3B.
Qiu-Lin Shang, Bin-Yu Wang, Yong-Qi Qiao et al.· 0 citations
QuantForge is introduced, a PTQ discovery system that records competing explanations, selects controls that distinguish them, and checks that successor code implements the resulting conclusions, improving the discovery of transferable PTQ algorithms by turning controlled evidence into subsequent program changes.
Qiu-Lin Shang, Zhou-Tong Wu, Jie Hu et al.· 0 citations
Diffusion Transformers (DiTs) achieve strong video generation quality, but their dense spatiotemporal self-attention scales quadratically with sequence length and quickly becomes the dominant inference bottleneck. Sparse low-rank hybrids alleviate this cost by combining a local sparse branch with a global compressed br...
Ze-Kun Zhang, Yi-Xiang Cai, Yu-Xi Liu et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.