How to Train Your Latent Diffusion Language Model Jointly With the Latent Space

• URL: http://arxiv.org/abs/2605.07933v1
• Date: Fri, 08 May 2026 16:05:19 GMT
• Title: How to Train Your Latent Diffusion Language Model Jointly With the Latent Space
• Abstract summary: Latent diffusion models offer an attractive alternative to discrete diffusion for non-autoregressive text generation.<n>We present the Latent Diffusion Language Model (LDLM), in which the latent encoder, diffusion model, and decoder are trained jointly.<n>On OpenWebText and LM1B, LDLM achieves better generation performance than existing discrete and continuous diffusion language models.
• Score: 76.9057986588963
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Latent diffusion models offer an attractive alternative to discrete diffusion for non-autoregressive text generation by operating on continuous text representations and denoising entire sequences in parallel. The major challenge in latent diffusion modeling is constructing a suitable latent space. In this work, we present the Latent Diffusion Language Model (LDLM), in which the latent encoder, diffusion model, and decoder are trained jointly. LDLM builds its latent space by reshaping the representations of a pre-trained language model with a trainable encoder, yielding latents that are easy to both denoise and decode into tokens. We show that naive joint training produces a low-quality diffusion model, and propose a simple training recipe consisting of an MSE decoder loss, diffusion-to-encoder warmup, adaptive timestep sampling, and decoder-input noise. Ablations show that each component substantially impacts generation performance. On OpenWebText and LM1B, LDLM achieves better generation performance than existing discrete and continuous diffusion language models while being $2{\text -}13\times$ faster, indicating that jointly learning the latent space is a key step toward making latent diffusion competitive for text generation.

Related papers

• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling [60.04959047453115]
  AURORA-LM is a continuous-latent language model that separates the construction of a decodable text representation from the modeling of its distribution.<n>AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization.
  arXiv  Detail & Related papers  (2026-08-03T17:59:50Z)
• DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling [51.39835770694148]
  Diffusion language models intrinsically fail to capture correlations between decoded tokens.<n>We propose DiLaDiff, a variant of masked diffusion language models with three components.
  arXiv  Detail & Related papers  (2026-05-22T13:15:59Z)
• TextLDM: Language Modeling with Continuous Latent Diffusion [89.69255520673248]
  Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos.<n>We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification.
  arXiv  Detail & Related papers  (2026-05-08T13:54:34Z)
• CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think [17.27394520177311]
  CoDAR is a two-stage framework that keeps diffusion entirely continuous in an embedding space while learning a strong, context-conditional discretizer.<n>Experiments on LM1B and OpenWebText demonstrate that CoDAR substantially improves generation quality over latent diffusion.
  arXiv  Detail & Related papers  (2026-03-03T03:05:15Z)
• LLaDA-o: An Effective and Length-Adaptive Omni Diffusion Model [77.66516875262963]
  We present textbfLLaDA-o, an effective and length-adaptive omni diffusion model for multimodal understanding and generation.<n>Building on MoD, we introduce a data-centric length adaptation strategy that enables flexible-length decoding in multimodal settings.<n>Experiments show that LLaDA-o achieves state-of-the-art performance among omni-diffusion models on multimodal understanding and generation benchmarks.
  arXiv  Detail & Related papers  (2026-03-01T12:05:06Z)
• Segment-Level Diffusion: A Framework for Controllable Long-Form Generation with Diffusion Language Models [12.446047799880587]
  Token-level diffusion doesn't model word-order dependencies explicitly.<n>Passage-level diffusion struggles with learning robust representations for long-form text.<n>We propose Segment-Level Diffusion, a framework that enhances diffusion-based text generation.
  arXiv  Detail & Related papers  (2024-12-15T22:47:44Z)
• Multimodal Latent Language Modeling with Next-Token Diffusion [111.93906046452125]
  Multimodal generative models require a unified approach to handle both discrete data (e.g., text and code) and continuous data (e.g., image, audio, video)<n>We propose Latent Language Modeling (LatentLM), which seamlessly integrates continuous and discrete data using causal Transformers.
  arXiv  Detail & Related papers  (2024-12-11T18:57:32Z)
• LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation? [10.72249123249003]
  We revisit diffusion models, highlighting their capacity for holistic context modeling and parallel decoding. We introduce a novel architecture, LaDiC, which utilizes a split BERT to create a dedicated latent space for captions. LaDiC achieves state-of-the-art performance for diffusion-based methods on the MS dataset with 38.2 BLEU@4 and 126.2 CIDEr.
  arXiv  Detail & Related papers  (2024-04-16T17:47:16Z)
• A Cheaper and Better Diffusion Language Model with Soft-Masked Noise [62.719656543880596]
  Masked-Diffuse LM is a novel diffusion model for language modeling, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. We demonstrate that our Masked-Diffuse LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency.
  arXiv  Detail & Related papers  (2023-04-10T17:58:42Z)
• Latent Diffusion for Language Generation [26.620353485679892]
  Recent attempts to adapt diffusion to language have presented diffusion as an alternative to existing language models. We demonstrate that encoder-decoder language models can be utilized to efficiently learn high-quality language autoencoders. We validate the effectiveness of our approach for unconditional, class-conditional, and sequence-to-sequence language generation.
  arXiv  Detail & Related papers  (2022-12-19T13:57:06Z)
• Semantic-Conditional Diffusion Networks for Image Captioning [116.86677915812508]
  We propose a new diffusion model based paradigm tailored for image captioning, namely Semantic-Conditional Diffusion Networks (SCD-Net) In SCD-Net, multiple Diffusion Transformer structures are stacked to progressively strengthen the output sentence with better visional-language alignment and linguistical coherence. Experiments on COCO dataset demonstrate the promising potential of using diffusion models in the challenging image captioning task.
  arXiv  Detail & Related papers  (2022-12-06T16:08:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.