Multilinguality in Hybrid Attention LLMs
This work presents a first study of how hybrid attention impacts the multilinguality of LLMs, showing that cross-lingual representations in hybrid models develop in patterns tied to the ordering of recurrent and full-attention layers.