dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models

• URL: http://arxiv.org/abs/2603.18806v1
• Date: Thu, 19 Mar 2026 11:55:52 GMT
• Title: dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models
• Abstract summary: Diffusion Large Language Models (dLLMs) introduce a new paradigm for language generation.<n>In this work, we aim to improve the policy optimization for dLLMs by reducing the cost of the trajectory probability calculation.<n>We evaluate dTRPO on 7B dLLMs across instruction-following and reasoning benchmarks.
• Score: 71.26391195427878
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Diffusion Large Language Models (dLLMs) introduce a new paradigm for language generation, which in turn presents new challenges for aligning them with human preferences. In this work, we aim to improve the policy optimization for dLLMs by reducing the cost of the trajectory probability calculation, thereby enabling scaled-up offline policy training. We prove that: (i) under reference policy regularization, the probability ratio of the newly unmasked tokens is an unbiased estimate of that of intermediate diffusion states, and (ii) the probability of the full trajectory can be effectively estimated with a single forward pass of a re-masked final state. By integrating these two trajectory reduction strategies into a policy optimization objective, we propose Trajectory Reduction Policy Optimization (dTRPO). We evaluate dTRPO on 7B dLLMs across instruction-following and reasoning benchmarks. Results show that it substantially improves the core performance of state-of-the-art dLLMs, achieving gains of up to 9.6% on STEM tasks, up to 4.3% on coding tasks, and up to 3.0% on instruction-following tasks. Moreover, dTRPO exhibits strong training efficiency due to its offline, single-forward nature, and achieves improved generation efficiency through high-quality outputs.

Related papers

• A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models [9.133661763917374]
  We formulate reinforcement learning (RL) in continuous time with discrete state spaces and possibly arbitrary action spaces.<n>We develop a complete continuous-time RL framework for fine-tuning score-based discrete diffusion models.
  arXiv  Detail & Related papers  (2026-07-16T03:19:53Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design [45.80068602880684]
  We provide a systematic analysis of the RL design space by disentangling three factors: policy-gradient objectives, likelihood estimators, and rollout sampling schemes.<n>We show that adopting an evidence lower bound (ELBO) based model likelihood estimator, computed only from the final generated sample, is the dominant factor enabling effective, efficient, and stable RL optimization.
  arXiv  Detail & Related papers  (2026-02-04T15:36:42Z)
• IIB-LPO: Latent Policy Optimization via Iterative Information Bottleneck [20.113524065146674]
  Iterative Information Bottleneck (IIB-LPO) is a novel approach that shifts exploration from statistical perturbation of token to topological branching of reasoning trajectories.<n>IIB-LPO achieves state-of-the-art performance, surpassing prior methods by margins of up to 5.3% in accuracy and 7.4% in diversity metrics.
  arXiv  Detail & Related papers  (2026-01-09T15:46:40Z)
• BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clipping [69.74252624161652]
  We propose BAlanced Policy Optimization with Adaptive Clipping (BAPO)<n>BAPO dynamically adjusts clipping bounds to adaptively re-balance positive and negative contributions, preserve entropy, and stabilize RL optimization.<n>On AIME 2024 and AIME 2025 benchmarks, our 7B BAPO model surpasses open-source counterparts such as SkyWork-OR1-7B.
  arXiv  Detail & Related papers  (2025-10-21T12:55:04Z)
• Ring-lite: Scalable Reasoning via C3PO-Stabilized Reinforcement Learning for LLMs [51.21041884010009]
  Ring-lite is a Mixture-of-Experts (MoE)-based large language model optimized via reinforcement learning (RL)<n>Our approach matches the performance of state-of-the-art (SOTA) small-scale reasoning models on challenging benchmarks.
  arXiv  Detail & Related papers  (2025-06-17T17:12:34Z)
• MDPO: Multi-Granularity Direct Preference Optimization for Mathematical Reasoning [0.0]
  We propose the Multi-Granularity Direct Preference Optimization (MDPO) method, optimizing the mathematical reasoning of Large Language Models (LLMs)<n>We conduct experiments on the open-source models Qwen2 and Llama3, achieving improvements of 1.7% and 1.2% on the GSM8K dataset, and 2.3% and 1.2% on the MATH dataset.<n>We also provide a pipeline for constructing MDPO training data that is simple and does not require manual annotation costs.
  arXiv  Detail & Related papers  (2025-05-30T08:42:14Z)
• Accelerating RL for LLM Reasoning with Optimal Advantage Regression [52.0792918455501]
  We propose a novel two-stage policy optimization framework that directly approximates the optimal advantage function.<n>$A$*-PO achieves competitive performance across a wide range of mathematical reasoning benchmarks.<n>It reduces training time by up to 2$times$ and peak memory usage by over 30% compared to PPO, GRPO, and REBEL.
  arXiv  Detail & Related papers  (2025-05-27T03:58:50Z)
• Towards Efficient Exact Optimization of Language Model Alignment [93.39181634597877]
  Direct preference optimization (DPO) was proposed to directly optimize the policy from preference data. We show that DPO derived based on the optimal solution of problem leads to a compromised mean-seeking approximation of the optimal solution in practice. We propose efficient exact optimization (EXO) of the alignment objective.
  arXiv  Detail & Related papers  (2024-02-01T18:51:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.