SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Results show that looping can improve Transformers even under budget matching, offering a practical recipe that turns depth reuse into measurable gains.
2 papers indexed here
We haven’t gathered this author’s papers yet. Follow them and we’ll fetch their work.
Not the right person? Other researchers publish under this name.
Results show that looping can improve Transformers even under budget matching, offering a practical recipe that turns depth reuse into measurable gains.
Together, the perplexity analysis indicates improved continuation predictability, while the controlled pre-training experiments suggest that this augmentation can improve model performance without changing the standard pre-training objective.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.