Diffusion LMs Can Approximate Optimal Infilling Lengths Implicitly

• URL: http://arxiv.org/abs/2602.00476v1
• Date: Sat, 31 Jan 2026 03:00:21 GMT
• Title: Diffusion LMs Can Approximate Optimal Infilling Lengths Implicitly
• Abstract summary: Diffusion language models (DLMs) provide a bidirectional generation framework naturally suited for infilling.<n>In this paper, we reveal that DLMs possess an inherent ability to discover the correct infilling length.<n>Our training-free method textbfCAL enables DLMs to approximate the optimal length through an efficient search before formal decoding.
• Score: 16.576341843767352
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion language models (DLMs) provide a bidirectional generation framework naturally suited for infilling, yet their performance is constrained by the pre-specified infilling length. In this paper, we reveal that DLMs possess an inherent ability to discover the correct infilling length. We identify two key statistical phenomena in the first-step denoising confidence: a local \textit{Oracle Peak} that emerges near the ground-truth length and a systematic \textit{Length Bias} that often obscures this signal. By leveraging this signal and calibrating the bias, our training-free method \textbf{CAL} (\textbf{C}alibrated \textbf{A}daptive \textbf{L}ength) enables DLMs to approximate the optimal length through an efficient search before formal decoding. Empirical evaluations demonstrate that CAL improves Pass@1 by up to 47.7\% over fixed-length baselines and 40.5\% over chat-based adaptive methods in code infilling, while boosting BLEU-2 and ROUGE-L by up to 8.5\% and 9.9\% in text infilling. These results demonstrate that CAL paves the way for robust DLM infilling without requiring any specialized training. Code is available at https://github.com/NiuHechang/Calibrated_Adaptive_Length.

Related papers

• Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models [78.02669593911784]
  Diffusion language models (DLMs) have emerged as a promising alternative to the auto-regressive paradigm.<n>We propose PILL (Probing-based InfiLling with preset-Length-free decoding), an efficient infilling method for DLMs.
  arXiv  Detail & Related papers  (2026-09-02T04:52:01Z)
• Metropolis-Adjusted Diffusion Models [39.6979566317926]
  Sampling from score-based diffusion models incurs bias due to time discretisation and the approximation of the score function.<n>We introduce the first exact method for adjusting Langevin corrections in diffusion models, based on a two-coin Bernoulli factory algorithm.
  arXiv  Detail & Related papers  (2026-05-10T16:57:51Z)
• Infeasibility Aware Large Language Models for Combinatorial Optimization [12.884852956582762]
  We propose an infeasibility-aware framework that combines certifiable dataset construction, supervised fine-tuning, and LLM-assisted downstream search.<n>Experiments show that our fine-tuned model improves overall accuracy by up to 30% over GPT-5.2; meanwhile LLM-guided warm starts provide up to $2times$ speedup.
  arXiv  Detail & Related papers  (2026-04-01T23:02:44Z)
• $\ abla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• Predicting LLM Output Length via Entropy-Guided Representations [13.351384070796747]
  We introduce a lightweight framework that reuses the main model's internal hidden states for efficient length prediction.<n>Our framework features two core components: 1) Entropy-Guided Token Pooling (EGTP), which uses on-the-fly activations and token entropy for highly accurate static prediction.
  arXiv  Detail & Related papers  (2026-02-12T10:49:04Z)
• Lookahead Path Likelihood Optimization for Diffusion LLMs [31.01208893976334]
  We introduce path log-likelihood (Path LL), a trajectory-conditioned objective that strongly correlates with downstream accuracy and enables principled selection of unmasking paths.<n>To optimize Path LL at inference time, we propose POKE, an efficient value estimator that predicts the expected future Path LL of a partial decoding trajectory.<n>We then integrate this lookahead signal into POKE-SMC, a Sequential Monte Carlo-based search framework for dynamically identifying optimal unmasking paths.
  arXiv  Detail & Related papers  (2026-02-03T13:12:41Z)
• LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training [45.74983991122073]
  Large language models (LLMs) experience significant performance degradation when the input exceeds the pretraining context window.<n>Recent studies mitigate this problem by remapping OOD positions into the in-distribution range with fixed mapping strategies.<n>We propose Length-aware Multi-grained Positional Scaling (LaMPE), a training-free method that fully utilizes the model's effective context window.
  arXiv  Detail & Related papers  (2025-08-04T11:22:13Z)
• Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Models [74.15250326312179]
  Diffusion Large Language Models offer efficient parallel generation and capable global modeling.<n>The dominant application ofDLLMs is hindered by the need for a statically predefined generation length.<n>We introduce DAEDAL, a novel training-free denoising strategy that enables Dynamic Adaptive Length Expansion.
  arXiv  Detail & Related papers  (2025-08-01T17:56:07Z)
• SpeLLM: Character-Level Multi-Head Decoding [6.474939955469084]
  SpeLLM is a method that decouples input and output vocabularies by predicting character-level strings through multiple output heads.<n>In SpeLLM, each of the $k$ linear heads predicts a single character simultaneously, enabling the model to represent a much larger output space.<n>Our experiments with four pre-trained LLMs show their SpeLLM variants achieve competitive performance on downstream tasks while reducing runtime by 5.1%.
  arXiv  Detail & Related papers  (2025-07-22T08:07:06Z)
• ETT: Expanding the Long Context Understanding Capability of LLMs at Test-Time [4.737679362712655]
  ourmodelacronym(Extend at Test-Time) is a method for extending the context length of short context Transformer-based Language Models.<n>We evaluate ETT on LongBench by extending the context length of GPT-Large and Phi-2 up to 32 times, increasing from 1k to 32k tokens.
  arXiv  Detail & Related papers  (2025-07-08T18:06:45Z)
• Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights [75.83625828306839]
  textbfDrag-and-Drop LLMs (textitDnD) eliminates per-task training by mapping a handful of unlabeled task prompts directly to LoRA weight updates.<n>A lightweight text encoder distills each prompt batch into condition embeddings, which are then transformed by a cascaded hyper-convolutional decoder into the full set of LoRA matrices.
  arXiv  Detail & Related papers  (2025-06-19T15:38:21Z)
• Scaling Test-Time Compute Without Verification or RL is Suboptimal [70.28430200655919]
  We show that finetuning LLMs with verifier-based (VB) methods based on RL or search is far superior to verifier-free (VF) approaches based on distilling or cloning search traces, given a fixed amount of compute/data budget.<n>We corroborate our theory empirically on both didactic and math reasoning problems with 3/8B-sized pre-trained LLMs, where we find verification is crucial for scaling test-time compute.
  arXiv  Detail & Related papers  (2025-02-17T18:43:24Z)
• Cool-Fusion: Fuse Large Language Models without Training [73.17551121242602]
  Cool-Fusion fuses the knowledge of source LLMs, which does not require training.<n>Experiments have been conducted across a variety of benchmark datasets.<n>On GSM8K, Cool-Fusion increases accuracy from three strong source LLMs by a significant margin of 17.4%.
  arXiv  Detail & Related papers  (2024-07-29T09:02:19Z)
• Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient [57.9629676017527]
  We propose an optimization-based structural pruning that learns the pruning masks in a probabilistic space directly by optimizing the loss of the pruned model.<n>We achieve this by learning an underlying Bernoulli distribution to sample binary pruning masks.<n>Experiments conducted on LLaMA, LLaMA-2, LLaMA-3, Vicuna, and Mistral models demonstrate the promising performance of our method in efficiency and effectiveness.
  arXiv  Detail & Related papers  (2024-06-15T09:31:03Z)
• Nearest Neighbor Speculative Decoding for LLM Generation and Attribution [87.3259169631789]
  Nearest Speculative Decoding (NEST) is capable of incorporating real-world text spans of arbitrary length into the LM generations and providing attribution to their sources.<n>NEST significantly enhances the generation quality and attribution rate of the base LM across a variety of knowledge-intensive tasks.<n>In addition, NEST substantially improves the generation speed, achieving a 1.8x speedup in inference time when applied to Llama-2-Chat 70B.
  arXiv  Detail & Related papers  (2024-05-29T17:55:03Z)
• From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers [52.199303258423306]
  We propose a novel density loss that encourages higher activation sparsity in pre-trained models. Our proposed method, textbfDEFT, can consistently reduce activation density by up to textbf44.94% on RoBERTa$_mathrmLarge$ and by textbf53.19% (encoder density) and textbf90.60% (decoder density) on Flan-T5$_mathrmXXL$.
  arXiv  Detail & Related papers  (2024-02-02T21:25:46Z)
• Scaling Sparse Fine-Tuning to Large Language Models [67.59697720719672]
  Large Language Models (LLMs) are difficult to fully fine-tune due to their sheer number of parameters. We propose SpIEL, a novel sparse finetuning method which maintains an array of parameter indices and the deltas of these parameters relative to their pretrained values. We show that SpIEL is superior to popular parameter-efficient fine-tuning methods like LoRA in terms of performance and comparable in terms of run time.
  arXiv  Detail & Related papers  (2024-01-29T18:43:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.