Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?

• URL: http://arxiv.org/abs/2605.15855v1
• Date: Fri, 15 May 2026 11:14:13 GMT
• Title: Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?
• Abstract summary: Most studies apply RL to the full denoising trajectory, making it computationally costly and weakening preference alignment.<n>We propose AdaScope, an RL-enhanced plug-in that improves generation quality while reducing computational cost.<n>Compared with state-of-the-art methods, AdaScope improves performance by 66% while cutting computational cost by 59%.
• Score: 30.438505697106496
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Despite strong image-generation performance, diffusion models' reconstruction objectives limit alignment with human preferences. RL enables such alignment through explicit rewards. However, most studies apply RL to the full denoising trajectory, making it computationally costly and weakening preference alignment, i.e., doing more but achieving less. We observe that the impact of RL fine-tuning varies significantly across denoising stages. In the early stage, image structures are unstable and distant from the final reward signal. Applying RL at this stage leads to delayed rewards and action-reward mismatching, resulting in high variance and inefficient updates. Conversely, in the later stage, reward gains saturate, and continued training tends to overfit local details, intensifying reward hacking. To tackle these challenges, we propose AdaScope, an RL-enhanced plug-in that improves generation quality while reducing computational cost. Specifically, AdaScope adaptively identifies the optimal intervention timing for RL by perceiving the structural evolution and semantic consistency during denoising, and dynamically terminates training once the denoising converges and reward gains saturate. As a result, it achieves a rare 'dual benefit': a reduction in computational costs alongside a significant performance improvement. We offer theoretical grounds for the design of AdaScope. Compared with state-of-the-art methods, AdaScope improves performance by 66% while cutting computational cost by 59%.

Related papers

• PAST: Prompt-Adaptive Sampling Termination for Efficient Diffusion Model [11.180362540642689]
  PAST enhances computational efficiency of existing RL fine-tuning methods by up to 66.7%.<n>It also improves preference optimization quality by up to 29.5% through its dual adaptive regulation mechanism.
  arXiv  Detail & Related papers  (2026-08-07T04:35:26Z)
• Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models [11.180362540642689]
  Reinforcement learning for preference alignment in diffusion models is promising but limited by reward sparsity.<n>We propose Stage-Guided Per-Step Optimization (SGPO) for diffusion models.<n>Our method achieves 26.7% average gains in generative quality and 36.7% higher convergence speed.
  arXiv  Detail & Related papers  (2026-08-07T03:41:58Z)
• Exposure-Based Reinforcement Learning to Rank [28.688480043859894]
  Reinforcement learning (RL) methods can optimize (almost) any ranking goal, e.g., from precision or discounted cumulative gain to fairness-of-exposure or ranking distillation.<n>Existing methods reach computational efficiency through custom gradient algorithms, but they are very complex to implement and often clash with auto-differentiation.
  arXiv  Detail & Related papers  (2026-07-21T04:11:20Z)
• Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models [58.3184497327891]
  Distilled autoregressive (AR) video models enable efficient streaming generation but frequently misalign with human visual preferences.<n>We present Astrolabe, an efficient online reinforcement learning framework tailored for distilled AR models.
  arXiv  Detail & Related papers  (2026-03-17T18:32:18Z)
• LACONIC: Length-Aware Constrained Reinforcement Learning for LLM [29.383977698780374]
  LACONIC is a reinforcement learning method that enforces a target token budget during training.<n>It preserves or improves pass@1 while reducing output length by over 50%.<n>It maintains out-of-domain performance on general knowledge and multilingual benchmarks with 44% fewer tokens.
  arXiv  Detail & Related papers  (2026-02-16T05:09:40Z)
• Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates [53.3717573880076]
  We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates.<n>JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly.<n>Experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods.
  arXiv  Detail & Related papers  (2026-01-26T14:16:51Z)
• From Supervision to Exploration: What Does Protein Language Model Learn During Reinforcement Learning? [76.288870982181]
  Protein language models (PLMs) have advanced computational protein science through large-scale pretraining and scalable architectures.<n> reinforcement learning (RL) has broadened exploration and enabled precise multi-objective optimization in protein design.<n>We ask if RL improves sampling efficiency and, more importantly, if it reveals capabilities not captured by supervised learning.
  arXiv  Detail & Related papers  (2025-10-02T01:31:10Z)
• BroRL: Scaling Reinforcement Learning via Broadened Exploration [88.69554867685243]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key ingredient for unlocking complex reasoning capabilities in large language models.<n>Recent work ProRL has shown promise in scaling RL by increasing the number of training steps.<n>We investigate a complementary paradigm for scaling RL, BroR-Lincreasing the number of rollouts per example to hundreds.
  arXiv  Detail & Related papers  (2025-10-01T17:59:02Z)
• Step-level Reward for Free in RL-based T2I Diffusion Model Fine-tuning [23.02076024811612]
  Recent advances in text-to-image (T2I) diffusion model fine-tuning leverage reinforcement learning (RL) to align generated images with learnable reward functions.<n>Existing approaches reformulate denoising as a Markov decision process for RL-driven optimization.<n>We propose a credit assignment framework that dynamically distributes dense rewards across denoising steps.
  arXiv  Detail & Related papers  (2025-05-25T15:43:54Z)
• Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards [52.90573877727541]
  reinforcement learning (RL) has been considered for diffusion model fine-tuning.<n>RL's effectiveness is limited by the challenge of sparse reward.<n>$textB2text-DiffuRL$ is compatible with existing optimization algorithms.
  arXiv  Detail & Related papers  (2025-03-14T09:45:19Z)
• Attention-Driven Training-Free Efficiency Enhancement of Diffusion Models [29.863953001061635]
  Diffusion Models (DMs) have exhibited superior performance in generating high-quality and diverse images. Existing works mainly adopt a retraining process to enhance DM efficiency. We introduce the Attention-driven Training-free Efficient Diffusion Model (AT-EDM) framework that leverages attention maps to perform run-time pruning of redundant tokens.
  arXiv  Detail & Related papers  (2024-05-08T17:56:47Z)
• PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models [13.313186665410486]
  Reward finetuning has emerged as a promising approach to aligning foundation models with downstream objectives. Existing reward finetuning methods are limited by their instability in large-scale prompt datasets. We propose Proximal Reward Difference Prediction (PRDP) to enable stable black-box reward finetuning for diffusion models.
  arXiv  Detail & Related papers  (2024-02-13T18:58:16Z)
• Towards Practical Lipreading with Distilled and Efficient Models [57.41253104365274]
  Lipreading has witnessed a lot of progress due to the resurgence of neural networks. Recent works have placed emphasis on aspects such as improving performance by finding the optimal architecture or improving generalization. There is still a significant gap between the current methodologies and the requirements for an effective deployment of lipreading in practical scenarios. We propose a series of innovations that significantly bridge that gap: first, we raise the state-of-the-art performance by a wide margin on LRW and LRW-1000 to 88.5% and 46.6%, respectively using self-distillation.
  arXiv  Detail & Related papers  (2020-07-13T16:56:27Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.