Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models

• URL: http://arxiv.org/abs/2603.12893v1
• Date: Fri, 13 Mar 2026 10:54:09 GMT
• Title: Finite Difference Flow Optimization for RL Post-Training of Text-to-Image Models
• Abstract summary: Reinforcement learning (RL) has become a standard technique for post-training diffusion-based image synthesis models.<n>We propose an online RL variant that reduces the variance in the model updates by sampling paired trajectories and pulling the flow velocity in the direction of the more favorable image.<n>Our method converges faster and yields higher output quality and prompt alignment than previous approaches.
• Score: 41.56292507255188
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) has become a standard technique for post-training diffusion-based image synthesis models, as it enables learning from reward signals to explicitly improve desirable aspects such as image quality and prompt alignment. In this paper, we propose an online RL variant that reduces the variance in the model updates by sampling paired trajectories and pulling the flow velocity in the direction of the more favorable image. Unlike existing methods that treat each sampling step as a separate policy action, we consider the entire sampling process as a single action. We experiment with both high-quality vision language models and off-the-shelf quality metrics for rewards, and evaluate the outputs using a broad set of metrics. Our method converges faster and yields higher output quality and prompt alignment than previous approaches.

Related papers

• Know Your Step: Faster and Better Alignment for Flow Matching Models via Step-aware Advantages [6.470160796651034]
  We propose a novel framework for training flow matching text to image models into efficient few step generators well aligned with human preferences.<n>We show that TAFS GRPO achieves strong performance in few step text to image generation and significantly improves the alignment of generated images with human preferences.
  arXiv  Detail & Related papers  (2026-02-02T03:32:00Z)
• Enhancing Diffusion-based Restoration Models via Difficulty-Adaptive Reinforcement Learning with IQA Reward [93.04811239892852]
  Reinforcement Learning (RL) has recently been incorporated into diffusion models.<n>In this paper, we investigate how to effectively integrate RL into diffusion-based restoration models.
  arXiv  Detail & Related papers  (2025-11-03T14:57:57Z)
• AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning [56.71089466532673]
  We propose AR-GRPO, an approach to integrate online RL training into autoregressive (AR) image generation models.<n>We conduct comprehensive experiments on both class-conditional (i.e., class-to-image) and text-conditional (i.e., text-to-image) image generation tasks.<n>Our results show consistent improvements across various evaluation metrics.
  arXiv  Detail & Related papers  (2025-08-09T10:37:26Z)
• Fine-Tuning Next-Scale Visual Autoregressive Models with Group Relative Policy Optimization [1.1510009152620668]
  Fine-tuning pre-trained generative models with Reinforcement Learning (RL) has emerged as an effective approach for aligning outputs with human preferences.<n>We show that RL-based fine-tuning is both efficient and effective for VAR models, benefiting particularly from their fast inference speeds.
  arXiv  Detail & Related papers  (2025-05-29T10:45:38Z)
• Policy Optimized Text-to-Image Pipeline Design [73.9633527029941]
  We introduce a novel reinforcement learning-based framework for text-to-image generation.<n>Our approach first trains an ensemble of reward models capable of predicting image quality scores directly from prompt-workflow combinations.<n>We then implement a two-phase training strategy: initial vocabulary training followed by GRPO-based optimization.
  arXiv  Detail & Related papers  (2025-05-27T17:50:47Z)
• Flow-GRPO: Training Flow Matching Models via Online RL [80.62659379624867]
  We propose Flow-GRPO, the first method to integrate online policy reinforcement learning into flow matching models.<n>Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation into an equivalent Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps.
  arXiv  Detail & Related papers  (2025-05-08T17:58:45Z)
• Weak Supervision Dynamic KL-Weighted Diffusion Models Guided by Large Language Models [0.0]
  We present a novel method for improving text-to-image generation by combining Large Language Models with diffusion models.<n>Our approach incorporates semantic understanding from pre-trained LLMs to guide the generation process.<n>Our method significantly improves both the visual quality and alignment of generated images with text descriptions.
  arXiv  Detail & Related papers  (2025-02-02T15:43:13Z)
• Learning Efficient and Effective Trajectories for Differential Equation-based Image Restoration [59.744840744491945]
  In this paper, we reformulate the trajectory optimization of this kind of method, focusing on enhancing both reconstruction quality and efficiency.<n>To mitigate the considerable computational burden associated with iterative sampling, we propose cost-aware trajectory distillation.<n>We fine-tune a foundational diffusion model (FLUX) with 12B parameters by using our algorithms, producing a unified framework for handling 7 kinds of image restoration tasks.
  arXiv  Detail & Related papers  (2024-10-07T07:46:08Z)
• RL for Consistency Models: Faster Reward Guided Text-to-Image Generation [15.238373471473645]
  We propose a framework for fine-tuning consistency models viaReinforcement Learning (RL) Our framework, called Reinforcement Learning for Consistency Model (RLCM), frames the iterative inference process of a consistency model as an RL procedure. Comparing to RL finetuned diffusion models, RLCM trains significantly faster, improves the quality of the generation measured under the reward objectives, and speeds up the inference procedure by generating high quality images with as few as two inference steps.
  arXiv  Detail & Related papers  (2024-03-25T15:40:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.