Unlocking the Potential of Diffusion Language Models through Template Infilling

• URL: http://arxiv.org/abs/2510.13870v1
• Date: Mon, 13 Oct 2025 12:33:41 GMT
• Title: Unlocking the Potential of Diffusion Language Models through Template Infilling
• Authors: Junhoo Lee, Seungyeon Kim, Nojun Kwak,
• Abstract summary: Diffusion Language Models (DLMs) have emerged as a promising alternative to Autoregressive Language Models.<n>We propose Template Infilling (TI), a tailored conditioning methodology for DLMs' generation process.<n>We show that TI provides additional advantages in multi-token generation settings, enabling effective speedup while maintaining generation quality.
• Score: 33.69224085914102
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion Language Models (DLMs) have emerged as a promising alternative to Autoregressive Language Models, yet their inference strategies remain limited to prefix-based prompting inherited from the autoregressive paradigm. In this paper, we propose Template Infilling (TI), a tailored conditioning methodology for DLMs' generation process. Unlike conventional prefix prompting, TI first generates a structural template for the target response, then fills in the masked segments. To enhance the flexibility of this structural control, we introduce Dynamic Segment Allocation (DSA), which adaptively adjusts segment lengths based on generation confidence. We demonstrate the effectiveness of our approach on mathematical reasoning and code generation benchmarks, achieving consistent improvements of 17.01$\%$p over baseline. Furthermore, we show that TI provides additional advantages in multi-token generation settings, enabling effective speedup while maintaining generation quality.

Related papers

• SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment [21.557846771500426]
  SemPA boosts the sentence representations while preserving the generative ability of LLMs via semantic preference alignment.<n>We establish a formal connection between DPO and contrastive learning under the Plackett-Luce model framework.
  arXiv  Detail & Related papers  (2026-01-08T16:19:24Z)
• OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generation [91.45421429922506]
  OneCAT is a unified multimodal model that seamlessly integrates understanding, generation, and editing.<n>Our framework eliminates the need for external components such as Vision Transformers (ViT) or vision tokenizer during inference.
  arXiv  Detail & Related papers  (2025-09-03T17:29:50Z)
• Improving Text Style Transfer using Masked Diffusion Language Models with Inference-time Scaling [37.795834398730555]
  Masked diffusion language models (MDMs) have recently gained traction as a viable generative framework for natural language.<n>We propose a verifier-based inference-time scaling method that aids in finding a better candidate generation during the denoising process of the MDM.<n>Our experiments demonstrate the application of MDMs for standard text-style transfer tasks and establish MDMs as a better alternative to autoregressive language models.
  arXiv  Detail & Related papers  (2025-08-14T18:01:22Z)
• Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking [15.052244821404079]
  We introduce Adaptive-Free Guidance (A-CFG), a novel method that tailors unconditional input by leveraging the model's predictive confidence.<n>A-CFG focuses on areas of ambiguity leading to more effective guidance.<n> Experiments on diverse language generation benchmarks show that A-CFG yields substantial improvements over standard CFG.
  arXiv  Detail & Related papers  (2025-05-26T16:40:22Z)
• Latent Thought Models with Variational Bayes Inference-Time Computation [52.63299874322121]
  Latent Thought Models (LTMs) incorporate explicit latent thought vectors that follow an explicit prior model in latent space.<n>LTMs demonstrate superior sample and parameter efficiency compared to autoregressive models and discrete diffusion models.
  arXiv  Detail & Related papers  (2025-02-03T17:50:34Z)
• Chunk-Distilled Language Modeling [25.238256586953487]
  Chunk-Distilled Language Modeling (CD-LM) is an approach to text generation that addresses two challenges in current large language models (LLMs)<n>Our method combines deep network-based LLMs with a straightforward retrieval module, which allows the generation of multi-token text chunks at a single decoding step.
  arXiv  Detail & Related papers  (2024-12-31T08:32:15Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• Harnessing the Plug-and-Play Controller by Prompting [12.705251690623495]
  This paper introduces a novel method for flexible attribute control in text generation using pre-trained language models (PLMs) The proposed approach aims to enhance the fluency of generated text by guiding the generation process with PPCs.
  arXiv  Detail & Related papers  (2024-02-06T17:18:25Z)
• FLIP: Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Prediction [49.510163437116645]
  Click-through rate (CTR) prediction plays as a core function module in personalized online services. Traditional ID-based models for CTR prediction take as inputs the one-hot encoded ID features of tabular modality. Pretrained Language Models(PLMs) has given rise to another paradigm, which takes as inputs the sentences of textual modality. We propose to conduct Fine-grained feature-level ALignment between ID-based Models and Pretrained Language Models(FLIP) for CTR prediction.
  arXiv  Detail & Related papers  (2023-10-30T11:25:03Z)
• PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model [37.2192243883707]
  We propose PLANNER, a model that combines latent semantic diffusion with autoregressive generation to generate fluent text. Results on semantic generation, text completion and summarization show its effectiveness in generating high-quality long-form text.
  arXiv  Detail & Related papers  (2023-06-05T01:36:39Z)
• Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study [76.52997424694767]
  We present an in-depth empirical study of keyphrase extraction and keyphrase generation using pre-trained language models. We show that PLMs have competitive high-resource performance and state-of-the-art low-resource performance. Further results show that in-domain BERT-like PLMs can be used to build strong and data-efficient keyphrase generation models.
  arXiv  Detail & Related papers  (2022-12-20T13:20:21Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• Progressive Generation of Long Text with Pretrained Language Models [83.62523163717448]
  Large-scale language models (LMs) pretrained on massive corpora of text, such as GPT-2, are powerful open-domain text generators. It is still challenging for such models to generate coherent long passages of text, especially when the models are fine-tuned to the target domain on a small corpus. We propose a simple but effective method of generating text in a progressive manner, inspired by generating images from low to high resolution.
  arXiv  Detail & Related papers  (2020-06-28T21:23:05Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.