This work reports interpretability experiments for a tokenized autoregressive Transformer agent in the Game of Hidden Rules (GOHR), a compact two-rule task in which both hidden rules map object shapes to target buckets, but with different permutations.
Open-weight language models from different families exhibit complementary capabilities, motivating their consolidation into a compact student through on-policy distillation (OPD). However, full-vocabulary OPD typically assumes a shared tokenizer, while existing cross-tokenizer methods may discard teacher probability ma...
Hao Wang, Kun Yuan, Wenlin Zhong et al.· arXiv.org· 5 citations· ⚡1
AsymSpec, an asymmetric speculative decoding framework that breaks this symmetry: a lightweight drafter reads the full input while the large verifier operates on the compressed view, is proposed, showing that asymmetric context access yields substantial gains precisely when compression discards critical reasoning signa...
Sheng Liang, Yong-Yue Zhang, Nathanael Brian et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.