Dor Tirosh

Pretraining Latent Information Feedback Transformers with Teacher Supervision

Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for information to flow downward across generation steps is the decoded token. This narrow channel forces models to recompute intermediate results and to discard alternative contin...