Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment

• URL: http://arxiv.org/abs/2605.06885v1
• Date: Thu, 07 May 2026 19:35:48 GMT
• Title: Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment
• Abstract summary: Diffusion language models (DLMs) have recently demonstrated capabilities that complement standard autoregressive (AR) models.<n>We ask whether the internal representation geometry learned by next-token prediction can be explicitly preserved during AR-to-DLM conversion.<n>We introduce REPR-ALIGN, a representation alignment objective that adapts a bidirectional masked diffusion model to reuse representations from a pretrained AR model.
• Score: 46.75006425771645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion language models (DLMs) have recently demonstrated capabilities that complement standard autoregressive (AR) models, particularly in non-sequential generation and bidirectional editing. Although recent work has shown that pretrained autoregressive checkpoints can be converted into diffusion language models, existing recipes primarily transfer parameters through continued denoising training with objective- and attention-level modifications. We instead ask whether the internal representation geometry learned by next-token prediction can be explicitly preserved during AR-to-DLM conversion. We hypothesize that much of the semantic structure learned by AR pretraining can transfer across generation orders, and thus DLM training should be viewed as relearning the decoding path rather than relearning language representations. To investigate this, we introduce REPR-ALIGN, a representation alignment objective that adapts a bidirectional masked diffusion model to reuse representations from a pretrained AR model of identical architecture. Concretely, we align the hidden states of the DLM to the frozen AR model at every layer using cosine similarity, while optimizing the standard masked denoising objective. This simple alignment, with no adapters and no architectural changes beyond the attention mask, yields up to 4x training acceleration in our setting and is particularly effective in low-data regimes. Our results suggest that linguistic representations can transfer across generation order, and that representation alignment provides a simple and effective technique for training diffusion language models. Code is available at https://github.com/pengzhangzhi/Open-dLLM.

Related papers

• Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation [50.27099285181793]
  We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs)<n>We introduce an On-Policy Diffusion Language Model (OPDLM) in which On-Policy Distillation (OPD) is employed for ARLM-to-DLM transformation.
  arXiv  Detail & Related papers  (2026-06-04T20:58:08Z)
• SimSD: Simple Speculative Decoding in Diffusion Language Models [61.33773959352141]
  Diffusion large language models (dLLMs) offer faster inference through parallel or blockwise decoding.<n>Their masked language modeling formulation remains incompatible with standard token-level speculative decoding.<n>We propose a speculative decoding algorithm for dLLMs, named SimSD, that equips dLLMs with temporally valid token-level contexts.<n>Our method achieves up to 7.46x higher decoding throughput while maintaining and even improving average generation quality.
  arXiv  Detail & Related papers  (2026-06-01T17:46:46Z)
• Machine Unlearning for Masked Diffusion Language Models [14.122398562530032]
  Masked diffusion language models (MDLMs) generate text by iteratively denoising masked positions in parallel.<n>We propose Masked Diffusion Unlearning (MDU), the first unlearning framework for MDLMs.<n>MDU minimizes a forward KL divergence from the prompt-conditional prediction to a prompt-masked unconditional anchor at every masked response position.
  arXiv  Detail & Related papers  (2026-05-18T11:54:11Z)
• TextLDM: Language Modeling with Continuous Latent Diffusion [89.69255520673248]
  Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos.<n>We propose TextLDM, which transfers the visual latent diffusion recipe to text generation with minimal architectural modification.
  arXiv  Detail & Related papers  (2026-05-08T13:54:34Z)
• Skip to the Good Part: Representation Structure & Inference-Time Layer Skipping in Diffusion vs. Autoregressive LLMs [39.20818218608441]
  Autoregressive (AR) language models form representations incrementally through left-to-right prediction, whereas diffusion language models (dLLMs) are trained via full-sequence denoising.<n>We find that diffusion objectives result in different, more hierarchical abstractions with substantial early-layer redundancy and reduced recency bias.<n>We introduce a static, task-agnostic inference-time layer-skipping method requiring no architectural changes or KV-cache sharing.<n>Native dLLMs achieve up to 18.75% FLOPs reduction while preserving over 90% performance on reasoning and code generation benchmarks, whereas AR models degrade sharply under comparable skipping
  arXiv  Detail & Related papers  (2026-03-08T05:31:52Z)
• Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed [76.49335677120031]
  Diffusion language models (dLMs) have emerged as a promising paradigm that enables parallel, non-autoregressive generation.<n>We study AR-to-dLM conversion to transform pretrained AR models into efficient dLMs that excel in speed while preserving AR models' task accuracy.
  arXiv  Detail & Related papers  (2025-12-16T04:12:17Z)
• Anchored Diffusion Language Model [39.17770765212062]
  We introduce the Anchored Diffusion Language Model (ADLM), a novel framework that predicts distributions over important tokens via an anchor network.<n>ADLM significantly improves test perplexity on LM1B and OpenWebText, achieving up to 25.4% gains over prior DLMs.<n>It also surpasses AR models in MAUVE score, which marks the first time a DLM generates better human-like text than an AR model.
  arXiv  Detail & Related papers  (2025-05-24T01:34:14Z)
• Unleashing the Potential of Large Language Models for Text-to-Image Generation through Autoregressive Representation Alignment [32.11442595627763]
  Autoregressive Representation Alignment (ARRA) is a new training framework that unlocks global-coherent text-to-image generation in autoregressive LLMs without architectural modifications.<n>ARRA aligns LLM's hidden states with visual representations from external visual foundational models via a global visual alignment loss and a hybrid token, [object Object]<n>Extensive experiments validate ARRA's plug-and-play versatility.
  arXiv  Detail & Related papers  (2025-03-10T13:49:28Z)
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models [105.70889434492143]
  Diffusion Language Models (DLMs) have emerged as a promising new paradigm for text generative modeling.<n>We show that we can convert AR models ranging from 127M to 7B parameters into diffusion models DiffuGPT and DiffuLLaMA, using less than 200B tokens for training.<n>Our experimental results reveal that these models outperform earlier DLMs and are competitive with their AR counterparts.
  arXiv  Detail & Related papers  (2024-10-23T14:04:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.