PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention

Zhengtao Yao Run-Hao Li Xu-Peng Chen Jia-Yi Cheng Chen-Qian Le Michael Yue Jesson Wang Siheng Wang Guang Yang Haoyan Xu Chen Wei Zheng-Qing Yuan Youran Shen Yanfang Ye Junhao Dong

Jul 2026 · arXiv.org · Vol abs/2607.25157 · 0 citations · 25 references

Computer Science

TL;DR

PreDiff-LM preserves causal attention within the observed prompt while allowing full bidirectional attention within the masked target, position hybrid attention as a complementary mechanism for adapting pretrained causal backbones, while making explicit the remaining quality and inference-efficiency gaps to optimized AR models.

Abstract

Discrete masked diffusion language models support bidirectional generation and infilling, but adapting pretrained autoregressive (AR) transformers requires reconciling causal pretraining with bidirectional denoising. We study this problem at the level of attention rather than claiming AR-weight reuse itself as novel. PreDiff-LM preserves causal attention within the observed prompt while allowing full bidirectional attention within the masked target. Under a matched GPT-2 Medium, WikiText-103, 90K-step setup, this hybrid mask improves unconditional perplexity from 34.1 to 28.7 and MAUVE from 0.71 to 0.78 over uniform bidirectional attention with the same AR initialization. Attention adaptation also composes with a DiffuGPT-style objective adaptation, reaching 26.9 perplexity. Pretrained initialization reduces the steps required to reach perplexity below 50 from about 350K to 8K, although a compute-matched fine-tuned AR model remains stronger at equal scale (18.9 versus 28.7). Beyond perplexity, PreDiff-LM improves repetition, distributional quality, four zero-shot downstream tasks, and human preference over prior diffusion baselines. The results position hybrid attention as a complementary mechanism for adapting pretrained causal backbones, while making explicit the remaining quality and inference-efficiency gaps to optimized AR models.

DOI arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

Retrofitting Linear Attention into Diffusion Language Models

This work introduces block-hybrid attention, which retains exact softmax attention within the active denoising block while applying linear attention over previous blocks, and shows that pretrained dLLMs can be efficiently linearized for faster inference.

Jinha Kim, Younghun Roh, Jaeyeon Kim · 0 citations

Jul 2026

UNIFUSION: Adapting Autoregressive Language Models into Discrete Diffusion under a Unified Reverse-Rate Objective

This work proposes a simple continual pre-training approach for directly adapting pretrained GPT2 checkpoints to uniform-noise diffusion, and establishes connections among SEDD, MDLM/GIDD, M2S, and Neural CTMC by expressing their conditional losses as a single generalized Kullback--Leibler objective over model reverse...

Xiaoyi Jiang, Jingyuan Li, Yi-Xuan Jiang et al. · arXiv.org · 0 citations

#natural language process... Preprint Aug 2026

Reducing Pretraining-Generation Mismatch in Diffusion Language Models

Prefix-Conditioned Diffusion (PCD), which samples a boundary, preserves the prefix, preserves the prefix, and denoises the suffix, is proposed, and improvements in reasoning and coding over native diffusion training are shown.

Xiao-Cheng Lu, Hua-Bin Liu, Song Guo et al. · 0 citations

Jul 2026

Trace-Based On-Policy Distillation for Masked Diffusion Language Models

Diffusion large language models (dLLMs) are a promising alternative to autoregressive generation. However, reasoning-oriented post-training for dLLMs remains challenging. Supervised fine-tuning (SFT) for dLLMs requires dense but often off-policy masked states, while reinforcement learning (RL) relies on sparse rewards...

Haolin Ren, Ziyang Huang, Chenhao Yuan et al. · arXiv.org · 0 citations

Preprint Aug 2026

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

This work systematically characterize how optimization hyperparameters, compute allocation, and architecture scale for MoE dLLMs, identifying quantitative differences from scaling trends previously reported for AR models.

Fengqi Zhu, Shaoxuan Xu, Jingyang Ou et al. · 1 citation

Jul 2026

Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models

This work studies how diffusion language models implement induction, a mechanism behind in-context learning in which the model finds a repeated context and copies the token that followed it, and compares attention-only AR models and absorbing-mask DLMs with matched architectures.

Andy Catruna, Emilian Radoi · arXiv.org · 0 citations