LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction

• URL: http://arxiv.org/abs/2608.05600v1
• Date: Thu, 06 Aug 2026 04:47:57 GMT
• Title: LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction
• Abstract summary: Flow-based generative models are typically sampled by solving a deterministic ordinary differential equation (ODE)<n>Existing GRPO methods for flow models replace the Euler-time ODE with aotropic differential equation (SDE) during training.<n>We introduce LC-GRPO, a flow-based GRPO framework with Langevin correction targeting the marginal distribution at the resulting timestep.
• Score: 59.32834240526981
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Flow-based generative models are typically sampled by solving a deterministic ordinary differential equation (ODE), whereas online reinforcement learning requires stochastic rollouts for policy exploration and optimization. Existing GRPO methods for flow models therefore replace the inference-time ODE with a stochastic differential equation (SDE) during training. Although the ODE and SDE share the same marginal distributions in continuous time, their finite-step discretizations can differ substantially. In particular, SDE rollouts often become blurry as the exploration noise increases, creating a mismatch between the samples used for reinforcement learning and those generated by the test-time ODE sampler. We introduce LC-GRPO, a flow-based GRPO framework with Langevin correction. Each rollout transition first takes an inference-aligned ODE Euler step and then applies a stochastic Langevin correction targeting the marginal distribution at the resulting timestep. The required score is recovered directly from the flow velocity, requiring no additional score model, while the resulting transition remains an isotropic Gaussian with a tractable likelihood for policy optimization. We theoretically show that, under suitable conditions, one Langevin correction step reduces the Wasserstein error of an imperfect ODE Euler step. At a matched randomness level, we further show that the proposed transition can be more accurate than the standard Euler--Maruyama discretization of the reverse SDE. Experiments on SD3.5-Medium, FLUX.1-Dev, and HunyuanVideo demonstrate that LC-GRPO consistently improves reward optimization across text-to-image and text-to-video tasks, preserves generation quality, and substantially narrows the gap between stochastic training rollouts and deterministic test-time ODE inference.

Related papers

• Guided Flow Matching for Forward and Inverse PDE Problems with Sparse Observations: Algorithm and Theory [2.148615003900331]
  We present FM4PDE, a flow-matching generative framework that learns the joint distribution of PDE coefficients.<n>Experiments on static and time-dependent benchmark PDEs demonstrate competitive accuracy and faster inference than diffusion-based generative models.
  arXiv  Detail & Related papers  (2026-05-25T07:14:11Z)
• Precise: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models [56.67321805551389]
  Reinforcement learning (RL) has become an effective way to improve prompt alignment and perceptual quality in diffusion and flow-matching generators.<n>The sampler, controlling the exploration behavior and denoising dynamics, is thus part of the policy.<n>We propose a new sampler that balances effective exploration with stability.
  arXiv  Detail & Related papers  (2026-05-22T11:37:22Z)
• Variational Smoothing and Inference for SDEs from Sparse Data with Dynamic Neural Flows [2.9746075314244376]
  We develop a novel method for smoothing the posterior differential equations (SDEs) in partially observed systems.<n>We learn this conditional score using neural networks trained to satisfy both the governing PDE and the observation-induced jump conditions.<n>The resulting score induces a posterior SDE with the same diffusion coefficient but a modified drift, enabling efficient posterior trajectory.
  arXiv  Detail & Related papers  (2026-05-07T02:47:06Z)
• Neural Stochastic Flows: Solver-Free Modelling and Inference for SDE Solutions [23.147474211347856]
  We introduce Neural Flows (NSFs) and their latent variants, which directly learn (latent) SDE transition laws.<n>Experiments on synthetic SDE simulations and on real-world tracking and video data show that NSFs maintain distributional accuracy comparable to numerical approaches.
  arXiv  Detail & Related papers  (2025-10-29T17:59:06Z)
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling [70.8832906871441]
  We study how to steer generation toward desired rewards without retraining the models.<n>Prior methods typically resample or filter within a single denoising trajectory, optimizing rewards step-by-step without trajectory-level refinement.<n>We introduce particle Gibbs sampling for diffusion language models (PG-DLM), a novel inference-time algorithm enabling trajectory-level refinement while preserving generation perplexity.
  arXiv  Detail & Related papers  (2025-07-11T08:00:47Z)
• Flow-GRPO: Training Flow Matching Models via Online RL [80.62659379624867]
  We propose Flow-GRPO, the first method to integrate online policy reinforcement learning into flow matching models.<n>Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation into an equivalent Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps.
  arXiv  Detail & Related papers  (2025-05-08T17:58:45Z)
• Gaussian Mixture Solvers for Diffusion Models [84.83349474361204]
  We introduce a novel class of SDE-based solvers called GMS for diffusion models. Our solver outperforms numerous SDE-based solvers in terms of sample quality in image generation and stroke-based synthesis.
  arXiv  Detail & Related papers  (2023-11-02T02:05:38Z)
• Adaptive Annealed Importance Sampling with Constant Rate Progress [68.8204255655161]
  Annealed Importance Sampling (AIS) synthesizes weighted samples from an intractable distribution. We propose the Constant Rate AIS algorithm and its efficient implementation for $alpha$-divergences.
  arXiv  Detail & Related papers  (2023-06-27T08:15:28Z)
• Diffusion Schr\"odinger Bridge with Applications to Score-Based Generative Modeling [24.46142828617484]
  Diffusion SB is an original approximation of the Iterative Proportional Fitting (IPF) procedure to solve the Schr"odinger Bridge problem. We present Diffusion SB, an original approximation of the Iterative Proportional Fitting (IPF) procedure to solve the SB problem, and provide theoretical analysis along with generative modeling experiments.
  arXiv  Detail & Related papers  (2021-06-01T17:34:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.