Score-Based One-step MeanFlow Policy Optimization

• URL: http://arxiv.org/abs/2605.23365v1
• Date: Fri, 22 May 2026 08:28:51 GMT
• Title: Score-Based One-step MeanFlow Policy Optimization
• Abstract summary: MeanFlow offers a promising alternative by learning an average velocity field that maps noise to data in a single network evaluation.<n>We propose Score-Based One-step MeanFlow Policy Optimization (SOM), an actor-critic algorithm that resolves this by constructing the target velocity field directly from the Q-function.<n>SOM achieves state-of-the-art performance on locomotion tasks with a single generation step, while substantially reducing both training and inference time compared to prior diffusion- and flow-matching-based policies.
• Score: 4.94481688445056
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion and flow matching have emerged as expressive policy classes in reinforcement learning, but their reliance on multi-step denoising imposes substantial computational overhead at inference time, which is particularly problematic in online RL. MeanFlow offers a promising alternative by learning an average velocity field that maps noise to data in a single network evaluation. However, MeanFlow typically requires samples from the target distribution to construct its target velocity field, which are unavailable in online RL. We propose Score-Based One-step MeanFlow Policy Optimization (SOM), an actor-critic algorithm that resolves this by constructing the target velocity field directly from the Q-function via score estimation and a probability flow ODE, thereby concentrating probability mass on high-value modes. In the fully online RL setting, SOM achieves state-of-the-art performance on locomotion tasks with a single generation step, while substantially reducing both training and inference time compared to prior diffusion- and flow-matching-based policies.

Related papers

• Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning [50.738952715864116]
  Expressive continuous control policies form the backbone of advances in scaling imitation learning for simulated and real robot control.<n>We propose QGF (Q-Guided Flow), an RL algorithm that performs policy optimization entirely at test time.<n> Empirically, QGF outperforms prior test-time RL methods on single-task and goal-conditioned offline RL benchmarks.
  arXiv  Detail & Related papers  (2026-06-09T16:45:57Z)
• Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models [61.74572554094633]
  We argue that ratio clipping is structurally ill-suited for flow models.<n>We propose Flow-DPPO, which replaces ratio clipping with a divergence proximal constraint.<n> Experiments show that Flow-DPPO achieves higher rewards with better KL-proximal efficiency.
  arXiv  Detail & Related papers  (2026-06-09T15:59:57Z)
• Aligning Flow Map Policies with Optimal Q-Guidance [50.514994916864275]
  Flow map policies are designed for fast action generation by learning to take arbitrary-size jumps.<n>We derive FLOW MAP Q-GUIDANCE (FMQ), a principled closed-form learning target that is optimal for adapting offline flow map policies under a critic-guided trust-region constraint.<n>FMQ achieves state-of-the-art performance in offline-to-online RL, outperforming the previous one-step policy MVP by a relative improvement of 21.3% on the average success rate.
  arXiv  Detail & Related papers  (2026-05-12T17:12:29Z)
• Mean Flow Policy Optimization [30.210126355529372]
  MeanFlow models are a class of few-step flow-based generative models.<n>We optimize MeanFlow policies under the maximum entropy RL framework.<n>Experiments on MuJoCo and DeepMind Control Suite benchmarks demonstrate that our method achieves performance comparable to or exceeding current diffusion-based baselines.
  arXiv  Detail & Related papers  (2026-04-16T06:59:52Z)
• Riemannian MeanFlow for One-Step Generation on Manifolds [54.09734511705173]
  Flow Matching enables simulation-free training of generative models.<n>MeanFlow can be extended to manifold-valued generation where velocities lie in location-dependent tangent spaces.<n>Experiments on spheres, tori, and SO(3) demonstrate competitive one-step sampling with improved quality-efficiency trade-offs and substantially reduced sampling cost.
  arXiv  Detail & Related papers  (2026-03-11T12:41:46Z)
• Reverse Flow Matching: A Unified Framework for Online Reinforcement Learning with Diffusion and Flow Policies [4.249024052507976]
  We propose a unified framework, reverse flow matching (RFM), which rigorously addresses the problem of training diffusion and flow models without direct target samples.<n>By adopting a reverse inferential perspective, we formulate the training target as a posterior mean estimation problem given an intermediate noisy sample.<n>We show that existing noise-expectation and gradient-expectation methods are two specific instances within this broader class.
  arXiv  Detail & Related papers  (2026-01-13T01:58:24Z)
• Generative Modeling with Continuous Flows: Sample Complexity of Flow Matching [60.37045080890305]
  We provide the first analysis of the sample complexity for flow-matching based generative models.<n>We decompose the velocity field estimation error into neural-network approximation error, statistical error due to the finite sample size, and optimization error due to the finite number of optimization steps for estimating the velocity field.
  arXiv  Detail & Related papers  (2025-12-01T05:14:25Z)
• One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow [56.13949180229929]
  We introduce a one-step generative policy for offline reinforcement learning that maps noise directly to actions via a residual reformulation of MeanFlow.<n>Our method achieves strong performance in both offline and offline-to-online reinforcement learning settings.
  arXiv  Detail & Related papers  (2025-11-17T06:34:17Z)
• SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modeling [9.936731043466699]
  Training expressive flow-based policies with off-policy reinforcement learning is notoriously unstable due to gradient pathologies in the multi-step action sampling process.<n>We trace this instability to a fundamental connection: the flow rollout is algebraically equivalent to a residual recurrent computation, making it susceptible to the same vanishing and exploding gradients as RNNs.<n>We develop a practical SAC-based algorithm, enabled by a noise-augmented rollout, that facilitates direct end-to-end training of these policies.
  arXiv  Detail & Related papers  (2025-09-30T04:21:20Z)
• MeanFlowSE: one-step generative speech enhancement via conditional mean flow [13.437825847370442]
  MeanFlowSE is a conditional generative model that learns the average velocity over finite intervals along a trajectory.<n>On VoiceBank-DEMAND, the single-step model achieves strong intelligibility, fidelity, and perceptual quality with substantially lower computational cost than multistep baselines.
  arXiv  Detail & Related papers  (2025-09-18T11:24:47Z)
• One-Step Flow Policy Mirror Descent [52.31612487608593]
  Flow Policy Mirror Descent (FPMD) is an online RL algorithm that enables 1-step sampling during flow policy inference.<n>Our approach exploits a theoretical connection between the distribution variance and the discretization error of single-step sampling in straight flow matching models.
  arXiv  Detail & Related papers  (2025-07-31T15:51:10Z)
• Efficient Regression-Based Training of Normalizing Flows for Boltzmann Generators [85.25962679349551]
  Boltzmann Generators (BGs) offer efficient sampling and likelihoods, but their training via maximum likelihood is often unstable and computationally challenging.<n>We propose Regression Training of Normalizing Flows (RegFlow), a novel and scalable-based training objective that bypasses the numerical instability and computational challenge of conventional maximum likelihood training.
  arXiv  Detail & Related papers  (2025-06-01T20:32:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.