Jul 2026· International journal of information and communication technology trends· 0 citations
TL;DR
HCC directly realizes the hierarchical-compression direction identified as future work by recent segmented-decoding methods, and is validated with controlled and real-text information-retention studies and with direct latency and memory measurements.
Abstract
Diffusion language models (DLMs) generate text by iteratively denoising an entire sequence in parallel, an appealing alternative to token-by-token autoregressive decoding. On long inputs, however, conditioning every denoising step on the full history costs time and memory that grow linearly with sequence length, and recent segmented decoders that compress each completed window into a small, fixed set of summary tokens still incur a total injected context that scales linearly with the number of windows. We propose Hierarchical Context Compression (HCC), which organizes per-window summaries into a suffix-dyadic tree of “summaries of summaries”: recent windows are kept at fine resolution while progressively older windows are covered by recursively merged, coarser summaries. HCC uses only two learned operators—a leaf compressor and a shared binary MERGE—applied through a segment-tree decomposition of the window history. We prove that HCC reduces the active context injected at each denoising step from O(n) to O(log n) summary vectors while still covering the entire history (no window is dropped), and we give a rate–distortion bound predicting graceful rather than abrupt degradation of far-context fidelity. As full-scale DLM training exceeds our single-CPU budget, we validate the mechanism with controlled and real-text information-retention studies and with direct latency and memory measurements. On histories of 64 windows, HCC retains 5–6× more recoverable information about past windows than a fixed recent-window budget of equal asymptotic footprint, and its measured per-step latency is up to 240× lower than uncompressed full-history attention at 4096 windows while remaining essentially flat as the history grows. HCC directly realizes the hierarchical-compression direction identified as future work by recent segmented-decoding methods.
Masked diffusion language models (MDLMs) are attractive for long-form generation because they denoise all positions with bidirectional context, but this advantage also creates a severe inference-time memory bottleneck: each denoising step requires dense attention over the active sequence. Recent sliding-window and speculative wrappers have shown that the bottleneck can be reduced without retraining the base model. However, fixed window sizes remain inefficient under heterogeneous documents, where some regions require broad context and others can be decoded safely with smaller memory footprints. This paper proposes Adaptive Window Diffusion Decoding (AWDD), a memory-budgeted inference framework that adjusts window size, overlap commitment, and context-summary capacity according to observable uncertainty in the partially masked sequence. AWDD uses a lightweight dependency-pressure score, a bounded summary buffer, and confidence-aware overlap commitment to preserve long-range terms while keeping peak memory below a fixed budget. We derive the memory and computational bounds of the scheduler and provide a reproducible CPU benchmark rather than unsupported large-model claims. On a controlled long-range dependency benchmark and a real-text sanity set extracted from the uploaded SW-SpeedDLM article, AWDD improves long-range entity reconstruction over fixed-window baselines while using substantially less peak memory than a large fixed window. These results support adaptive windowing as a practical direction for memory-constrained diffusion decoding and provide code, raw results, and document-level figures for replication.
N. Petrov, Rafael Costa, Clara Hoffmann· International Journal of Adv...· 0 citations
Diffusion Language Models (DLMs) exhibit strong parallel decoding capabilities by denoising multiple tokens in a single generation step. However, this parallelism comes with substantial computational overhead, as each step requires interactions with all suffix tokens. Existing methods typically reduce this cost by retaining only a local suffix window as a substitute for the full suffix. Despite their effectiveness, these methods overlook the structural heterogeneity across suffix regions and re-initialize suffix tokens with identical representations at each timestep. To this end, we propose a structured suffix modeling method for efficient DLM inference. Specifically, we divide the suffix into three regions, i.e., the local, middle, and tail regions, and retain different numbers of suffix tokens in each region according to their structural roles. Moreover, we incorporate the decoding results from the previous step into the suffix token representations at the current step, allowing them to carry evolving denoising information across generation steps. Notably, our method is training-free and orthogonal to several existing acceleration techniques, such as parallel decoding strategies and KV cache. Empirical results across multiple benchmarks on three DLMs demonstrate that our method can further accelerate DLM inference and improve performance in most cases. In particular, in long-sequence inference, our method achieves up to a \(72.81\times\) speedup when combined with other acceleration techniques. Our code is available at https://github.com/zifengcheng/SSM.
Zifeng Cheng, Keda Li, Zhiwei Jiang et al.· 0 citations
LaCache is proposed, a training-free acceleration framework that alleviates operator-level redundancy through lossless caching and mixed precision, and inegrates a per-group FP8 quantization strategy for FFN layers, tailored to step-dependent activation distributions across the diffusion process.
Xingru Chen, Zelang Liang, Yongjia Ma et al.· 0 citations
Masked diffusion language models (MDLMs) are attractive for long-form generation because they denoise many positions under bidirectional context rather than committing tokens strictly from left to right. The same property makes inference expensive: every reverse step may require dense attention over a long partially masked sequence. Recent sliding-window and speculative wrappers demonstrate that pretrained MDLMs can be accelerated without changing their weights, especially the SW-SpeedDLM framework, which combines segmented sliding-window denoising, cross-segment summary compression, and window-level speculative acceptance. This paper proposes Uncertainty-Aware Speculative Decoding for Diffusion Language Models (UASD-DLM), a complementary inference scheduler for long document generation. Instead of using a fixed speculation depth for every window, UASD-DLM estimates local predictive uncertainty from entropy, margin, and summary-conflict signals, then chooses both the draft depth and context refresh policy dynamically. Low-risk windows receive longer drafts, while high-risk windows are verified earlier to avoid wasted drafting and boundary errors. The acceptance step retains the same rejection-sampling correction as standard speculative decoding, so changing the draft length does not alter the verifier distribution. To avoid unsupported large-model claims, we provide a reproducible CPU proxy benchmark using real text extracted from the uploaded SW-SpeedDLM article and a seeded long-range dependency corpus. In this controlled setting, UASD-DLM improves the speed-memory trade-off over fixed-depth speculative decoding and recovers technical-term accuracy lost by summary-free windowing. The complete code, raw results, and Word manuscript accompany this submission.
Embedded Language Flows (ELF) rely primarily on full non-causal attention for iterative denoising, repeatedly incurring quadratic sequence-mixing cost at each sampling step. Gated Delta Networks (GDNs) provide an efficient recurrent alternative, but their standard causal formulation cannot directly capture the bidirectional context required by ELF. We introduce DeltaFlow, a noise-adaptive bidirectional GDN backbone for continuous language denoising. We study two variants: DeltaFlow-A, which alternates scan directions across layers, and DeltaFlow-P, which performs parallel forward and backward scans within each layer. We further introduce noise-adaptive memory control and scheduled Temporal State Consistency (TSC) to stabilize hidden representations across nearby noise levels. On OpenWebText, using a 32-step stochastic differential equation sampler, DeltaFlow-P reduces generated perplexity from 24.218 for the full-attention ELF baseline to 21.228 while maintaining comparable unigram entropy, with 36B training-token exposure compared with 45B for the baseline. In a denoiser-only benchmark, DeltaFlow-P achieves a 2.72x throughput speedup over the full-attention baseline at a sequence length of 16k. These results show that DeltaFlow is a promising alternative to dense attention for efficient continuous language denoising.
Guangfu Guo, Xiaoqian Lu, Linsey Pang et al.· 0 citations
DARTree is introduced, a training-free speculative decoding method that extends a pretrained AR correction head from chains to trees, and achieves the highest average acceptance length and speedup in all four model--temperature configurations.
Tianyi Li, Yaxin Luo, Xinyi Shang et al.· 0 citations