ReFPO: Reflow Regularization for Flow Matching Policy Gradients

• URL: http://arxiv.org/abs/2606.21086v1
• Date: Fri, 19 Jun 2026 04:23:10 GMT
• Title: ReFPO: Reflow Regularization for Flow Matching Policy Gradients
• Abstract summary: We present a simple online RL method that adds explicit Reflow regularization to Flow Matching Policy Gradients.<n>We experimentally demonstrate that ReFPO improves average performance and discretization across GridWorld, MuJoCo Playground, and high-dimensional Humanoid Control tasks.
• Score: 58.32178725687043
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Reflow-regularized Flow Matching Policy Gradients (ReFPO), a simple online RL method that adds explicit Reflow regularization to FPO for efficient flow-based control. We uncover a key structural property: the gradient updates in Flow Matching Policy Gradients (FPO) can be interpreted as an implicit advantage-weighted Reflow process, providing a new geometric perspective on flow-based policy gradients. Building on this insight, ReFPO introduces an explicit geometric regularizer that can be implemented with a single line of code change without incurring additional computational overhead or auxiliary distillation stages. By synergizing advantage-guided updates with path rectification, our method reduces CFM proxy-ratio spikes, stabilizes PPO-style training, and enables high-fidelity one-step inference that often matches or exceeds multi-step performance. We experimentally demonstrate that ReFPO improves average performance and discretization robustness across GridWorld, MuJoCo Playground, and high-dimensional Humanoid Control tasks, providing a scalable and stable approach for generative policies in complex physical simulations.

Related papers

• Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models [61.74572554094633]
  We argue that ratio clipping is structurally ill-suited for flow models.<n>We propose Flow-DPPO, which replaces ratio clipping with a divergence proximal constraint.<n> Experiments show that Flow-DPPO achieves higher rewards with better KL-proximal efficiency.
  arXiv  Detail & Related papers  (2026-06-09T15:59:57Z)
• Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descent [53.828537014796574]
  Online off-policy reinforcement learning (RL) is shaped by two choices: the policy class and the update rule.<n>We propose MeanFlow Policies, a one-step generative policy class that maps noise to actions through a MeanFlow transformation.<n>Across seven MuJoCo benchmarks, S improves over Gaussian and generative baselines while retaining single-step inference efficiency.
  arXiv  Detail & Related papers  (2026-05-20T15:14:14Z)
• OGPO: Sample Efficient Full-Finetuning of Generative Control Policies [53.42266064673132]
  Generative control policies (GCPs) have emerged as effective parameterizations for robot learning.<n>This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient for finetuning GCPs.<n>OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control.
  arXiv  Detail & Related papers  (2026-05-04T18:36:40Z)
• PolicyFlow: Policy Optimization with Continuous Normalizing Flow in Reinforcement Learning [6.836651088754774]
  PolicyFlow is a novel on-policy CNF-based reinforcement learning algorithm.<n>It integrates expressive CNF policies with PPO-style objectives without requiring likelihood evaluation along the full flow path.<n>PolicyFlow approximates importance ratios using velocity field variations along a simple path, reducing computational overhead without compromising training stability.
  arXiv  Detail & Related papers  (2026-02-01T11:08:09Z)
• Iterative Refinement of Flow Policies in Probability Space for Online Reinforcement Learning [56.47948583452555]
  We introduce the Stepwise Flow Policy (SWFP) framework, founded on the key insight that discretizing the flow matching inference process via a fixed-step Euler scheme aligns it with the variational Jordan-Kinderlehrer-Otto principle from optimal transport.<n>SWFP decomposes the global flow into a sequence of small, incremental transformations between proximate distributions.<n>This decomposition yields an efficient algorithm that fine-tunes pre-trained flows via a cascade of small flow blocks, offering significant advantages.
  arXiv  Detail & Related papers  (2025-10-17T07:43:51Z)
• Flow Matching Policy Gradients [33.011978450108636]
  Flow Policy Optimization casts policy optimization as maximizing an advantage-weighted ratio computed from the conditional flow matching loss.<n>We show that FPO can train diffusion-style policies from scratch in a variety of continuous control tasks.<n>We find that flow-based models can capture multimodal action distributions and achieve higher performance than Gaussian policies, particularly in under-conditioned settings.
  arXiv  Detail & Related papers  (2025-07-28T17:59:57Z)
• Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximization [4.158255103170876]
  GFlowNets are a family of generative models that learn to sample objects proportional to a given reward function.<n>Recent results show a close relationship between GFlowNet training and entropy-regularized reinforcement learning problems.<n>We introduce a simple backward policy optimization algorithm that involves direct sequentially of the value function in an entropy-regularized Markov Decision Process.
  arXiv  Detail & Related papers  (2024-10-20T19:12:14Z)
• REBEL: Reinforcement Learning via Regressing Relative Rewards [59.68420022466047]
  We propose REBEL, a minimalist RL algorithm for the era of generative models.<n>In theory, we prove that fundamental RL algorithms like Natural Policy Gradient can be seen as variants of REBEL.<n>We find that REBEL provides a unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO.
  arXiv  Detail & Related papers  (2024-04-25T17:20:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.