Learnability-Informed Fine-Tuning of Diffusion Language Models

• URL: http://arxiv.org/abs/2605.22939v1
• Date: Thu, 21 May 2026 18:16:17 GMT
• Title: Learnability-Informed Fine-Tuning of Diffusion Language Models
• Abstract summary: We propose LIFT, an efficient SFT-based post-training algorithm for diffusion language models (DLMs)<n>LIFT learns easy tokens when most of the input is masked and hard tokens when more context is available, thus aligning the training with the information available at different diffusion time steps.<n>Our results show that LIFT outperforms existing SFT baselines across six reasoning benchmarks, achieving up to a 3x relative gain on AIME'24 and AIME'25.
• Score: 50.420674346943294
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We aim to improve the reasoning capabilities of diffusion language models (DLMs). While SFT is a popular post-training recipe for autoregressive models, its use in DLMs faces challenges and can even hurt performance, though the underlying causes remain understudied. Our analysis reveals that vanilla SFT overlooks learnability, namely what and when tokens are learned. Specifically, rare tokens are difficult to learn when most of the input is masked, whereas it is straightforward and thus of little value to learn common tokens when most of the input is unmasked. Motivated by our analysis, we propose LIFT, an efficient SFT-based post-training algorithm for DLMs. LIFT learns easy tokens when most of the input is masked and hard tokens when more context is available, thus aligning the training with the information available at different diffusion time steps. Our results show that LIFT outperforms existing SFT baselines across six reasoning benchmarks, achieving up to a 3x relative gain on AIME'24 and AIME'25. Our code is publicly available at https://github.com/divelab/LIFT.

Related papers

• Which Tokens Should SFT Actually Learn? A Token-Trimming Perspective on Mathematical Reasoning [42.13393879881966]
  Supervised fine-tuning (SFT) applies a uniform cross-entropy loss to all target tokens, even though different tokens provide unequal learning signals for mathematical reasoning.<n>We propose Trimmed Logit-Gap SFT (TrimSFT), a simple token-level reweighting method that scales the SFT loss according to the logit gap between the gold token and its strongest competitor.<n>We evaluate TrimSFT on six base models from the Llama, Qwen, and DeepMath families across five mathematical reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-09-09T04:46:15Z)
• Boosting Reinforcement Learning with Verifiable Rewards via Randomly Selected Few-Shot Guidance [117.2119290254454]
  We propose FEST, a demonstration-guided RLVR algorithm.<n>It attains compelling results with only 128 demonstrations randomly selected from an SFT dataset.<n>We find that three components are vital for the success: supervised signal, on-policy signal, and decaying weights on the few-shot SFT dataset.
  arXiv  Detail & Related papers  (2026-05-14T16:12:30Z)
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification [61.607788999847564]
  We present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM)<n>We reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model.<n>We propose Dynamic Fine-Tuning (DFT), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token.
  arXiv  Detail & Related papers  (2025-08-07T17:59:04Z)
• ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs [1.1834200163382398]
  ReGATE (Reference$-$Guided Adaptive Token Elision) is an adaptive token pruning method for accelerating MLLM training.<n>It matches the peak accuracy of standard training on MVBench up to 2$times$ faster, using only 35% of the tokens.
  arXiv  Detail & Related papers  (2025-07-29T01:07:09Z)
• StochasTok: Improving Fine-Grained Subword Understanding in LLMs [39.85256850592515]
  Subword-level understanding is integral to numerous tasks, including understanding multi-digit numbers, spelling mistakes, abbreviations, rhyming, and wordplay.<n>Current large language models (LLMs) still often struggle with seemingly simple subword-level tasks.<n>We introduce StochasTok, a simple, efficient tokenization scheme that randomly splits tokens during training, allowing LLMs to'see their internal structure'
  arXiv  Detail & Related papers  (2025-06-02T13:51:11Z)
• Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning [53.57895922042783]
  Large Language Models (LLMs) excel at reasoning and planning when trained on chainof-thought (CoT) data.<n>We propose a hybrid representation of the reasoning process, where we partially abstract away the initial reasoning steps using latent discrete tokens.
  arXiv  Detail & Related papers  (2025-02-05T15:33:00Z)
• Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection [49.15148871877941]
  Next-token distribution outputs offer a theoretically appealing approach for detection of large language models (LLMs)<n>We propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length.<n>PAWN shows competitive and even better performance in-distribution than the strongest baselines with a fraction of their trainable parameters.
  arXiv  Detail & Related papers  (2025-01-07T17:00:49Z)
• Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression [40.4998607679863]
  Large Language Models (LLMs) often suffer from catastrophic forgetting when post-pretrained or supervised fine-tuned (SFT) on domain-specific data. This paper focuses on TG-SFT, which can synthetically generate SFT data for the instruction tuning steps.
  arXiv  Detail & Related papers  (2024-06-17T09:17:40Z)
• Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Models [54.72004797421481]
  We conduct the first systematic study to explore a decoding strategy specialized in code generation. Inspired by the above findings, we propose a simple yet effective method: Adaptive Temperature (AdapT) sampling. Results show that AdapT sampling significantly outperforms state-of-the-art decoding strategy.
  arXiv  Detail & Related papers  (2023-09-06T06:27:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.