This work introduces Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models, and shows that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately unif...
Feng-Zhuo Zhang, Shu-Che Wang, Sheng-Gui Li et al.· 0 citations
RPS is proposed, a novel training-free decoding method that seeks mid-entropy positions as promising candidate pivots (where to decode), and determines their token assignment that yields the greatest downstream benefit via lookahead evaluation (what to decode).
Yu-Shi Ye, Xu Chen, Hao-Yun Jiang et al.· 1 citation
Spiking neural networks (SNNs) offer a path to energy-efficient language modeling through sparse encoding and event-driven computation, but training capable spiking language models from scratch remains difficult. A practical alternative is ANN-to-SNN migration through knowledge distillation (KD), where a pretrained art...
En-Qiao Lu, Xingrui Yu, Yi-Wei Fu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.