Skip to content

Hierarchical Continuous Diffusion Language Models

Oct 2026 · 0 citations · 43 references
Computer Science

TL;DR

H Hierarchical Continuous Diffusion Language Models (HC-DLM) is proposed, which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood.

Abstract

Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded. To address this, we propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood. In contrast to recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and feed back as a scaffold for the next latent update. On structured reasoning (Sudoku), mathematical planning (Countdown) and language modeling (LM1B), HC-DLM improves over discrete and continuous diffusion baselines at matched model size, in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B. Project page: https://hc-dlm.github.io/.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

ALoDLM: Adaptively Looped Diffusion Language Models

ALoDLM is introduced, which replaces uniform computation with token-adaptive latent recurrence, and retains fast parallel decoding, yielding a strong quality-efficiency trade-off among evaluated autoregressive and diffusion models under optimized inference engines.

Lian-Cheng Fang, Zhuo-Wei Li, Youngeun Kim et al. · 1 citation
#machine learning Preprint Oct 2026

Rethinking the Roles of Diffusion and Decoding: Predictive-State Flows for Parallel Language Generation

Predictive-State Flows (PSF) is introduced, which partitions a sequence into blocks and associates each block with a predictive state computed solely from the preceding context, and suggests that latent design for continuous DLMs should consider not only how compactly text is represented, but also how generative roles...

Jin-Jin He, Xin-Song Feng, Lu-Cheng Fu et al. · 0 citations
#natural language process... Conference Open access Sep 2026

DA-DLM: Explicitly Modeling Token Dependencies in Diffusion Language Models

DA-DLM is proposed, a model that adapts DAG-based dependency modeling to DLMs'iterative setting through a position-oriented DAG design, and consistently outperforms Block Diffusion, especially under fewer denoising steps, and matches autoregressive models while preserving the parallel generation advantage.

Peng-Yu Ji, Zi-Chen Zhang, Xiang Hu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Large Language Continuous Diffusion Models

Despite the success of discrete diffusion language models (dLMs) for fast parallel decoding, their non-smooth, high-dimensional space hinders trajectory steering for reasoning and inference acceleration. To overcome this, we present Sigma, the first large-scale (3B/8B) continuous dLM built on steerable, low-dimensional...

Zhi-Han Yang, Wei Guo, Jean-Marie Lemercier et al. · 0 citations
#machine learning Preprint Sep 2026

Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or One

Language generation is almost universally treated as a sequential process: autoregressive models emit one token at a time, while diffusion language models replace token-level seriality with a long trajectory of iterative refinement. In this work, we introduce PlaidQ, a 0.7B continuous diffusion language model for code...

Fred Zhangzhi Peng, Kai-Wen Zheng, An-Ru R. Zhang · 1 citation · ⚡1
#artificial intelligence Preprint Oct 2026

RL Post-Training Diffusion Language Models Along Their Own Generation Paths

Diffusion language models are a promising and more efficient alternative to autoregressive models. However, most RL post-training techniques are not easily adapted to these models. One issue is that diffusion language models generate answers through many intermediate states, deciding at each step which masked positions...

Vaibhav Singh, Eugene Belilovsky · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.