When Does Depth Matter For In-Context Learning? Adaptive Inference in Deep Transformers
The results suggest that depth and feedforward blocks enable transformers to implement adaptive inference, and this is advantageous when the distribution over latent variables has hierarchical structure.
R. Raj, G.Pallavi Reddy
· 0 citations