Autoregressive video generation models generate videos sequentially, with each new block conditioned on previously generated blocks. By training on the model's own rollouts, Self Forcing and its follow-up methods substantially mitigate exposure bias caused by the mismatch between training- and inference-time history di...
Chen-Xi Xia, Yi-Xuan Xu, Shan Liu et al.· 0 citations
This work derives data mixing scaling laws for three modalities: text, vision, and audio from Mixture-of-Experts (MoE) architectures, and provides a principled foundation for understanding the multimodal data mixtures needed to train frontier MLLMs.
Aditi Khandelwal, A. Tarun, Yi-Xuan Xu et al.· 0 citations
A causal interpretability framework is introduced that reveals that the tokens encoding semantics at the input need not be those that maintain them during generation, but also provides a causal view of internal mechanisms in diffusion transformers.