Full-bandwidth transformer
This work trains 1B-parameter full-bandwidth transformers on up to 400B tokens and finds that latent feedback improves validation loss, 5-shot language-model evaluation, math and coding generation, and instruction-tuned performance.