Experiments with two-layer and GPT-2 Transformers reproduce the predicted width advantage and show that unequal weights fail first on the states that receive the least weight, suggesting a important principle: keep reached states equally weighted, and restore equal weights as computation proceeds.
Large language models solve hard problems through intermediate computations across multi-step reasoning. Traditional chain-of-thought encodes these computations as tokens. Recent continuous and recurrent methods instead move partial computations into fixed-dimensional latent states, where a single thought can superpose...
The proposed mHC-based semantic encoder applies multiple residual streams and constrains their interaction by doubly stochastic (DS) mixing matrices to improve semantic/task performance, communication robustness, and convergence stability over residual and unconstrained HC baselines, while requiring no additional chann...
Jing-Wen Fu, Ming Xiao· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.