SuperFlow: Training Flow Matching Models with RL on the Fly

• URL: http://arxiv.org/abs/2512.17951v1
• Date: Wed, 17 Dec 2025 02:44:11 GMT
• Title: SuperFlow: Training Flow Matching Models with RL on the Fly
• Abstract summary: SuperFlow is an RL training framework for flow-based models that adjusts group sizes with variance-aware sampling.<n>It reaches promising performance while using only 5.4% to 56.3% of the original training steps.<n>It reduces training time by 5.2% to 16.7% without any architectural changes.
• Score: 40.46209466164144
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent progress in flow-based generative models and reinforcement learning (RL) has improved text-image alignment and visual quality. However, current RL training for flow models still has two main problems: (i) GRPO-style fixed per-prompt group sizes ignore variation in sampling importance across prompts, which leads to inefficient sampling and slower training; and (ii) trajectory-level advantages are reused as per-step estimates, which biases credit assignment along the flow. We propose SuperFlow, an RL training framework for flow-based models that adjusts group sizes with variance-aware sampling and computes step-level advantages in a way that is consistent with continuous-time flow dynamics. Empirically, SuperFlow reaches promising performance while using only 5.4% to 56.3% of the original training steps and reduces training time by 5.2% to 16.7% without any architectural changes. On standard text-to-image (T2I) tasks, including text rendering, compositional image generation, and human preference alignment, SuperFlow improves over SD3.5-M by 4.6% to 47.2%, and over Flow-GRPO by 1.7% to 16.0%.

Related papers

• RFMSR: Residual Flow Matching for Image Super-Resolution [43.28301144364581]
  Residual Flow Matching for Image Super-Resolution (RFMSR)<n>Phase I pretrains the velocity field via conditional flow matching, while Phase II applies end-to-end supervision to the single-step prediction.<n>RFMSR achieves comparable or even superior perceptual quality compared to state-of-the-art (SOTA) methods.
  arXiv  Detail & Related papers  (2026-07-14T13:19:36Z)
• PromptRL: Prompt Matters in RL for Flow-Based Image Generation [44.81148439118129]
  We present PromptRL, a framework that incorporates language models (LMs) as trainable prompt refinement agents directly within the flow-based reinforcement learning loop.<n>PromptRL achieves state-of-the-art performance across multiple benchmarks, obtaining scores of 0.97 on GenEval, 0.98 on OCR accuracy, and 24.05 on PickScore.
  arXiv  Detail & Related papers  (2026-02-01T18:31:06Z)
• Flow Straighter and Faster: Efficient One-Step Generative Modeling via MeanFlow on Rectified Trajectories [14.36205662558203]
  Rectified MeanFlow is a framework that models the mean velocity field along the rectified trajectory using only a single reflow step.<n>Experiments on ImageNet at 64, 256, and 512 resolutions show that Re-MeanFlow consistently outperforms prior one-step flow distillation and Rectified Flow methods in both sample quality and training efficiency.
  arXiv  Detail & Related papers  (2025-11-28T16:50:08Z)
• Decoupled MeanFlow: Turning Flow Models into Flow Maps for Accelerated Sampling [68.76215229126886]
  We introduce Decoupled MeanFlow, a simple decoding strategy that converts flow models into flow map models without architectural modifications.<n>Our method conditions the final blocks of diffusion transformers on the subsequent timestep, allowing pretrained flow models to be directly repurposed as flow maps.<n>On ImageNet 256x256 and 512x512, our models attain 1-step FID of 2.16 and 2.12, respectively, surpassing prior art by a large margin.
  arXiv  Detail & Related papers  (2025-10-28T14:43:48Z)
• DiffusionNFT: Online Diffusion Reinforcement with Forward Process [99.94852379720153]
  Diffusion Negative-aware FineTuning (DiffusionNFT) is a new online RL paradigm that optimize diffusion models directly on the forward process via flow matching.<n>DiffusionNFT is up to $25times$ more efficient than FlowGRPO in head-to-head comparisons, while being CFG-free.
  arXiv  Detail & Related papers  (2025-09-19T16:09:33Z)
• ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning [7.452631636467998]
  ReinFlow is an online reinforcement learning framework for continuous robotic control.<n>It injects learnable noise into a flow policy's deterministic path, converting the flow into a discrete-time Markov Process.<n>We benchmark ReinFlow in representative locomotion and manipulation tasks.
  arXiv  Detail & Related papers  (2025-05-28T08:17:16Z)
• Flow-GRPO: Training Flow Matching Models via Online RL [80.62659379624867]
  We propose Flow-GRPO, the first method to integrate online policy reinforcement learning into flow matching models.<n>Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation into an equivalent Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps.
  arXiv  Detail & Related papers  (2025-05-08T17:58:45Z)
• Guided Flows for Generative Modeling and Decision Making [55.42634941614435]
  We show that Guided Flows significantly improves the sample quality in conditional image generation and zero-shot text synthesis-to-speech. Notably, we are first to apply flow models for plan generation in the offline reinforcement learning setting ax speedup in compared to diffusion models.
  arXiv  Detail & Related papers  (2023-11-22T15:07:59Z)
• GMFlow: Learning Optical Flow via Global Matching [124.57850500778277]
  We propose a GMFlow framework for learning optical flow estimation. It consists of three main components: a customized Transformer for feature enhancement, a correlation and softmax layer for global feature matching, and a self-attention layer for flow propagation. Our new framework outperforms 32-iteration RAFT's performance on the challenging Sintel benchmark.
  arXiv  Detail & Related papers  (2021-11-26T18:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.