Safety alignment for Large Language Models is heavily focused on English with limited multilingual representation, particularly damaging low-resource language performance. Related open training datasets are often limited in size or do not follow region-localized harmful topics to broaden the scope of alignment beyond c...
Max Neuwinger, H. Yukhymenko, Imanol Schlag· 0 citations
This work derives data mixing scaling laws for three modalities: text, vision, and audio from Mixture-of-Experts (MoE) architectures, and provides a principled foundation for understanding the multimodal data mixtures needed to train frontier MLLMs.
Aditi Khandelwal, A. Tarun, Yi-Xuan Xu et al.· 1 citation
Small language models are often post-trained as students on reasoning traces from stronger teacher models to efficiently learn new skills. However, token-level imitation on traces that lie far outside the student's expected distribution often produces \textit{confident conflicts}, whereby the student is required to imi...
Juan Garc'ia Giraldo, Matteo Santelmo, E. Durech et al.· 0 citations
Tabular foundation models achieve strong performance by conditioning on labelled examples in context, but softmax attention limits their use on large datasets. Existing linear-time alternatives, however, are mostly causal, and their potential for tabular in-context learning (ICL) remains underexplored. To address this,...
David Schnurr, Felix Sarnthein, Thomas Hofmann et al.· arXiv.org· 0 citations
A comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2 is presented, making explicit how they differ in expressivity, memory decay, erase and write control, training throughput, and implementation complexity...
Tommaso Cerruti, Tim Rieder, George Rowlands et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.