This work observes that MDLM generation involves two decisions at each step: what tokens to place at each masked position and which positions to remask, and formalizes this as a two-stage action MDP, showing that the policy gradient naturally decomposes into a token term and a masking term.
Abstract
Reinforcement learning has proven effective for improving reasoning in large language models, but extending it to Masked Diffusion Language Models (MDLMs) remains challenging due to the intractability of the log-likelihood estimation. Existing approaches approximate this log-likelihood by modeling only the token predictions, ignoring the order in which positions are unmasked during generation. We observe that MDLM generation involves two decisions at each step: what tokens to place at each masked position and which positions to remask. We formalize this as a two-stage action MDP, showing that the policy gradient naturally decomposes into a token term and a masking term. Combining optimization of both terms leads to state-of-the-art outcomes on mathematical reasoning and coding benchmarks, with scores of 87.1% on GSM8K and 53.4% on MBPP.
Diffusion large language models (dLLMs) are a promising alternative to autoregressive generation. However, reasoning-oriented post-training for dLLMs remains challenging. Supervised fine-tuning (SFT) for dLLMs requires dense but often off-policy masked states, while reinforcement learning (RL) relies on sparse rewards...
dOPSD derives the teacher's privilege directly from the student's own denoising trajectory, evaluating masked positions using later, more-decoded steps of that same trajectory rather than an external label, so the teacher's advantage emerges from the model's own decoding process.
It is shown that score-conditioned In-Context Learning (ICL) admits a structural correspondence to policy gradient optimization, and an exact upper bound on the distribution shift induced by a bounded attention update is derived, yielding a trust-region-like analogy to KL-constrained policy optimization.
Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard approach for improving complex reasoning in Large Language Models (LLMs). However, entropy collapse remains a critical bottleneck, often leading to performance degradation. While typically attributed to reduced exploration, we identify a more s...
Cong Qin, Jiaye Lin, Xiaoliang Fu et al.· Proceedings of the 32nd ACM...· 0 citations
This work studies how diffusion language models implement induction, a mechanism behind in-context learning in which the model finds a repeated context and copies the token that followed it, and compares attention-only AR models and absorbing-mask DLMs with matched architectures.
Andy Catruna, Emilian Radoi· arXiv.org· 0 citations
We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional au...
DiffusionGemma Team Adrien Ali Taïga, James Assiene, Daniele Calandriello et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
