MetaState: Persistent Working Memory for Discrete Diffusion Language Models

• URL: http://arxiv.org/abs/2603.01331v1
• Date: Mon, 02 Mar 2026 00:16:35 GMT
• Title: MetaState: Persistent Working Memory for Discrete Diffusion Language Models
• Abstract summary: We introduce textbfMetaState, a lightweight recurrent augmentation that equips a frozen dLLM backbone with a persistent, fixed-size working memory.<n>On LLaDA-8B and Dream-7B, textbfMetaState introduces negligible trainable parameters while keeping the backbone frozen, and it consistently improves accuracy over frozen baselines.
• Score: 13.775010785119761
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discrete diffusion language models (dLLMs) generate text by iteratively denoising a masked sequence. Compared with autoregressive models, this paradigm naturally supports parallel decoding, bidirectional context, and flexible generation patterns. However, standard dLLMs condition each denoising step only on the current hard-masked sequence, while intermediate continuous representations are discarded after sampling and remasking. We refer to this bottleneck as the \textbf{Information Island} problem. It leads to redundant recomputation across steps and can degrade cross-step consistency. We address this limitation with \textbf{MetaState}, a lightweight recurrent augmentation that equips a frozen dLLM backbone with a persistent, fixed-size working memory that remains independent of sequence length. \textbf{MetaState} consists of three trainable modules: a cross-attention Mixer that reads backbone activations into memory slots, a GRU-style Updater that integrates information across denoising steps, and a cross-attention Injector that feeds the updated memory back into backbone activations. We train these modules with $K$-step unrolling to expose them to multi-step denoising dynamics during fine-tuning. On LLaDA-8B and Dream-7B, \textbf{MetaState} introduces negligible trainable parameters while keeping the backbone frozen, and it consistently improves accuracy over frozen baselines. These results demonstrate that persistent cross-step memory is an effective mechanism for bridging denoising steps and improving generation quality in discrete diffusion language models.

Related papers

• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling [60.04959047453115]
  AURORA-LM is a continuous-latent language model that separates the construction of a decodable text representation from the modeling of its distribution.<n>AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization.
  arXiv  Detail & Related papers  (2026-08-03T17:59:50Z)
• DeMTS: Denoising Trajectories as Multivariate Time Series for Hallucination Detection in Diffusion Language Models [59.414664850109155]
  Diffusion large language models (D-LLMs) have emerged as a promising paradigm for text generation.<n>D-LLMs remain vulnerable to hallucinations, where fluent outputs may contain factually incorrect or unsupported content.
  arXiv  Detail & Related papers  (2026-07-24T06:45:43Z)
• DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs [29.561307941602482]
  We show that a pretrained masked DLM can be lightly adapted to support continuous embedding-space denoising.<n>The adapted model supports continuous inference that evolves all positions jointly in embedding space.<n> DSL-LLaDA-SDE achieves the best ROUGE-1 on all four benchmarks.
  arXiv  Detail & Related papers  (2026-05-31T05:27:01Z)
• Learned Relay Representations for Forward-Thinking Discrete Diffusion Models [34.17541648016911]
  We propose Learned Relay Representations (Relay) to avoid a hard reset between denoising rounds.<n>Relay introduces a differentiable per-token channel that passes information between forward passes and is trained via truncated backpropagation through time.<n>We show that Relay can be scaled to state-of-the-art Diffusion Language Models (DLMs) and is seamlessly compatible with techniques like block diffusion and KV caching.
  arXiv  Detail & Related papers  (2026-05-21T18:53:22Z)
• Prefix-Adaptive Block Diffusion for Efficient Document Recognition [52.15352911463151]
  Block Diffusion Models (BDMs) support parallel generation, flexible-length output, and KV caching, making them promising for efficient document parsing.<n>We propose the Prefix- Block Diffusion Model (PA-BDM) which replaces intra-block bidirectional denoising with causal denoising from prefix to suffix.<n> Experiments show that the 3B PA-BDM achieves higher recognition scores on several benchmarks and improves inference throughput by 71.6% over the 2.5B MinerU-Diffusion.
  arXiv  Detail & Related papers  (2026-05-16T07:50:13Z)
• MemDLM: Memory-Enhanced DLM Training [56.40248490616793]
  Diffusion Language Models (DLMs) offer attractive advantages over Auto-Regressive (AR) models.<n>They suffer from a notable train-inference mismatch.<n>We propose MemDLM (Memory-Enhanced DLM), which embeds a simulated denoising process into training.
  arXiv  Detail & Related papers  (2026-03-23T17:39:56Z)
• DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention [2.7422645382944935]
  We present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only salient tokens.<n>Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput.
  arXiv  Detail & Related papers  (2026-03-09T07:02:01Z)
• Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inference [58.189320101488725]
  DLLMs promise fast non-autoregressive inference but suffer a severe quality-speed trade-off in parallel decoding.<n>We address this by integrating continuous representations into the discrete decoding process, as they preserve rich inter-position dependency.<n>We propose ReMix, a framework that introduces a novel Continuous Mixing State as an intermediate between the initial masked state and the final decoded token state.
  arXiv  Detail & Related papers  (2026-02-26T11:08:11Z)
• Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models [58.946955321428845]
  This work presents self-rewarding sequential Monte Carlo (SMC)<n>Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy.<n>We introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights.
  arXiv  Detail & Related papers  (2026-02-02T09:21:45Z)
• Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding [36.74241893088594]
  Diffusion Large Language Models (dLLMs) offer a compelling paradigm for natural language generation.<n>Recent works have accelerated inference via KV cache reuse or decoding, but overlook the intrinsic inefficiencies within the block-wise diffusion process.<n>We propose Streaming-dLLM, a training-free framework that streamlines inference across both spatial and temporal dimensions.
  arXiv  Detail & Related papers  (2026-01-25T17:36:04Z)
• Mosaic: Unlocking Long-Context Inference for Diffusion LLMs via Global Memory Planning and Dynamic Peak Taming [34.16016695663811]
  Diffusion-based large language models (dLLMs) have emerged as a promising paradigm, utilizing simultaneous denoising to enable global planning and iterative refinement.<n>Existing inference systems are ill-suited for this paradigm due to severe system inefficiencies.<n>We propose Mosaic, a memory-efficient inference system that shifts from local, static management to a global, dynamic paradigm.
  arXiv  Detail & Related papers  (2026-01-10T13:17:08Z)
• DynaPURLS: Dynamic Refinement of Part-aware Representations for Skeleton-based Zero-Shot Action Recognition [51.80782323686666]
  We introduce textbfDynaPURLS, a unified framework that establishes robust, multi-scale visual-semantic correspondences.<n>Our framework leverages a large language model to generate hierarchical textual descriptions that encompass both global movements and local body-part dynamics.<n>Experiments on three large-scale benchmark datasets, including NTU RGB+D 60/120 and PKU-MMD, demonstrate that DynaPURLS significantly outperforms prior art.
  arXiv  Detail & Related papers  (2025-12-12T10:39:10Z)
• D$^{3}$ToM: Decider-Guided Dynamic Token Merging for Accelerating Diffusion MLLMs [22.78575203353886]
  Diffusion-based multimodal large language models (Diffusion MLLMs) exhibit substantially slower inference than autoregressive models.<n>We propose D$3$ToM, a Decider-guided dynamic token merging method to accelerate inference in Diffusion MLLMs.<n>Experiments show that D$3$ToM accelerates inference while preserving competitive performance.
  arXiv  Detail & Related papers  (2025-11-15T16:24:12Z)
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling [87.34677262370924]
  Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token.<n>This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps.<n>We introduce Continuously Augmented Discrete Diffusion, a framework that augments the discrete state space with a paired diffusion in a continuous latent space.
  arXiv  Detail & Related papers  (2025-10-01T18:00:56Z)
• Sequential Diffusion Language Models [110.06562906987052]
  Diffusion language models (DLMs) have strong theoretical efficiency but are limited by fixed-length decoding and incompatibility with key-value caches.<n>We introduce Next Sequence Prediction (NSP), which unifies next-token and next-block prediction.<n>We propose Sequential Diffusion Language Model (SDLM), which can retrofit pre-trained autoregressive language models (ALMs) at minimal cost.
  arXiv  Detail & Related papers  (2025-09-28T17:59:15Z)
• Unifying Autoregressive and Diffusion-Based Sequence Generation [3.1853022872760186]
  We present significant extensions to diffusion-based sequence generation models, blurring the line with autoregressive language models.<n>We introduce hyperschedules, which assign distinct noise schedules to individual token positions.<n>Second, we propose two hybrid token-wise noising processes that interpolate between absorbing and uniform processes, enabling the model to fix past mistakes.
  arXiv  Detail & Related papers  (2025-04-08T20:32:10Z)
• Autoregressive Speech Synthesis without Vector Quantization [135.4776759536272]
  We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS)<n>MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition.<n>MELLE mitigates robustness issues by avoiding the inherent flaws of sampling vector-quantized codes.
  arXiv  Detail & Related papers  (2024-07-11T14:36:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.