Continuous Latent Diffusion Language Model

• URL: http://arxiv.org/abs/2605.06548v1
• Date: Thu, 07 May 2026 16:44:56 GMT
• Title: Continuous Latent Diffusion Language Model
• Abstract summary: Large language models have achieved remarkable success under the autoregressive paradigm.<n>Existing alternatives still struggle to jointly achieve generation efficiency, scalable representation learning, and effective global semantic modeling.<n>We propose Cola DLM, a hierarchical latent diffusion language model that frames text generation through hierarchical information decomposition.
• Score: 48.974403879186916
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models have achieved remarkable success under the autoregressive paradigm, yet high-quality text generation need not be tied to a fixed left-to-right order. Existing alternatives still struggle to jointly achieve generation efficiency, scalable representation learning, and effective global semantic modeling. We propose Cola DLM, a hierarchical latent diffusion language model that frames text generation through hierarchical information decomposition. Cola DLM first learns a stable text-to-latent mapping with a Text VAE, then models a global semantic prior in continuous latent space with a block-causal DiT, and finally generates text through conditional decoding. From a unified Markov-path perspective, its diffusion process performs latent prior transport rather than token-level observation recovery, thereby separating global semantic organization from local textual realization. This design yields a more flexible non-autoregressive inductive bias, supports semantic compression and prior fitting in continuous space, and naturally extends to other continuous modalities. Through experiments spanning 4 research questions, 8 benchmarks, strictly matched ~2B-parameter autoregressive and LLaDA baselines, and scaling curves up to about 2000 EFLOPs, we identify an effective overall configuration of Cola DLM and verify its strong scaling behavior for text generation. Taken together, the results establish hierarchical continuous latent prior modeling as a principled alternative to strictly token-level language modeling, where generation quality and scaling behavior may better reflect model capability than likelihood, while also suggesting a concrete path toward unified modeling across discrete text and continuous modalities.

Related papers

• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling [60.04959047453115]
  AURORA-LM is a continuous-latent language model that separates the construction of a decodable text representation from the modeling of its distribution.<n>AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization.
  arXiv  Detail & Related papers  (2026-08-03T17:59:50Z)
• Textual Belief States for World Models: Identifiable Representation Learning Under Strict Mediation [19.960113437887383]
  We show how to make strict mediation work in the text domain.<n>We then introduce textual latent states, which are discrete, interpretable, and variable-length.<n>Experiments on TextWorld and ScienceWorld show preserved one-step prediction accuracy alongside up to 57% gains in representation quality.
  arXiv  Detail & Related papers  (2026-06-26T03:26:02Z)
• Self-Evolving Spatial Reasoning in Vision Language Models via Geometric Logic Consistency [53.28076739140119]
  We propose a framework that enforces logical consistency in Vision-Language Models (VLMs) through geometric and linguistic duality operations.<n>SAGE is model-agnostic, data-efficient compared to prior GRPO methods, and can be applied as a lightweight post-training stage to any existing VLM.<n> Experiments on video and spatial reasoning benchmarks demonstrate consistent improvements over strong baselines.
  arXiv  Detail & Related papers  (2026-05-18T10:05:21Z)
• TextLDM: Language Modeling with Continuous Latent Diffusion [89.69255520673248]
  Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos.<n>We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification.
  arXiv  Detail & Related papers  (2026-05-08T13:54:34Z)
• Flow of Spans: Generalizing Language Models to Dynamic Span-Vocabulary via GFlowNets [54.06320619464273]
  Flow of SpanS (FOSS) is a principled GFlowNets framework for span generation.<n>FOSS constructs a dynamic span vocabulary by segmenting the retrieved text flexibly.<n>With specialized reward models, FoSS generates diverse, high-quality text.
  arXiv  Detail & Related papers  (2026-02-11T07:17:41Z)
• Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion [60.186310080523135]
  Bifurcation of generative modeling into autoregressive approaches for discrete data (text) and diffusion approaches for continuous data (images) hinders development of truly unified multimodal systems.<n>We propose textbfCoM-DAD, a novel probabilistic framework that reformulates multimodal generation as a hierarchical dual-process.<n>Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.
  arXiv  Detail & Related papers  (2026-01-07T16:21:19Z)
• Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Models [12.446047799880587]
  Token-level diffusion doesn't model word-order dependencies explicitly.<n>Passage-level diffusion struggles with learning robust representations for long-form text.<n>We propose Segment-Level Diffusion, a framework that enhances diffusion-based text generation.
  arXiv  Detail & Related papers  (2024-12-15T22:47:44Z)
• PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model [37.2192243883707]
  We propose PLANNER, a model that combines latent semantic diffusion with autoregressive generation to generate fluent text. Results on semantic generation, text completion and summarization show its effectiveness in generating high-quality long-form text.
  arXiv  Detail & Related papers  (2023-06-05T01:36:39Z)
• Model Criticism for Long-Form Text Generation [113.13900836015122]
  We apply a statistical tool, model criticism in latent space, to evaluate the high-level structure of generated text. We perform experiments on three representative aspects of high-level discourse -- coherence, coreference, and topicality. We find that transformer-based language models are able to capture topical structures but have a harder time maintaining structural coherence or modeling coreference.
  arXiv  Detail & Related papers  (2022-10-16T04:35:58Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.