Multilinguality in Hybrid Attention LLMs
This work presents a first study of how hybrid attention impacts the multilinguality of LLMs, showing that cross-lingual representations in hybrid models develop in patterns tied to the ordering of recurrent and full-attention layers.
Lucas Bandarkar, Jun-Ling Hu, Chen-Yuan Yang et al.
· 0 citations