This work identifies activation error, rather than weight error, as the dominant source of FP4 RL instability: weights can be synchronized and aligned by a shared quantization-dequantization path, whereas activations are recomputed online and error is amplified by the coarse E2M1 grid.
Zhengyang Zhuge, Hao Yu, Xin Wang et al.· arXiv.org· 0 citations
The architecture and the Muon optimizer together shift the optimal learning rate and batch size upwards, render batch-size warmup unnecessary, and substantially improve stability under stress tests.
Zihan Qiu, Zekun Wang, Xiao Li et al.· 4 citations· ⚡1
This work discovers that certain attention heads exhibit sequential consistency in their attention patterns, which can be persistently identified using a coefficient-of-variation-based algorithm, and proposes CateKV, a hybrid KV cache method that retains only critical token information for consistent heads, thereby reducing KV cache size and computational overhead.
Hao-Yun Jiang, Hao-Lin Li, Jian-Wei Zhang et al.· International Conference on...· 2 citations
H-Scale is a lightweight post-processing method for NVFP4 per-group scale refinement that selects hardware-valid group scales using a diagonal second-order proxy derived from calibration activations, thereby targeting layer output perturbation more directly.
Hao Yu, Zheng Li, Dayiheng Liu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
