CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL

• URL: http://arxiv.org/abs/2605.14274v1
• Date: Thu, 14 May 2026 02:18:58 GMT
• Title: CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL
• Abstract summary: We introduce a compositional constraint-based reward model for post-training embodied video generation models.<n>We propose CreFlow, a novel online RL framework with two key designs.
• Score: 56.969946199335716
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video generation models trained on heterogeneous data with likelihood-surrogate objectives can produce visually plausible rollouts that violate physical constraints in embodied manipulation. Although reinforcement-learning post-training offers a natural route to adapting VGMs, existing video-RL rewards often reduce each rollout to a low-level visual metric, whereas manipulation video evaluation requires logic-based verification of whether the rollout satisfies a compositional task specification. To fill this gap, we introduce a compositional constraint-based reward model for post-training embodied video generation models, which automatically formulates task requirements as a composition of Linear Temporal Logic constraints, providing faithful rewards and localized error information in generated videos. To achieve effective improvement in high-dimensional video generation using these reward signals, we further propose CreFlow, a novel online RL framework with two key designs: i) a credit-aware NFT loss that confines the RL update to reward-relevant regions, preventing perturbations to unrelated regions during post-training; and ii) a corrective reflow loss that leverages within-group positive samples as an explicit estimate of the correction direction, stabilizing and accelerating training. Experiments show that CreFlow yields reward judgments better aligned with human and simulator success labels than existing methods and improves downstream execution success by 23.8 percentage points across eight bimanual manipulation tasks.

Related papers

• RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy [24.413419742450913]
  RedFlow is a fine-grained offline reinforcement learning framework for flow-matching VLA policies.<n>RedFlow redirects failure experiences into action-level corrective supervision for flow-matching VLA policies.<n>It consistently outperforms state-of-the-art offline RL baselines.
  arXiv  Detail & Related papers  (2026-07-30T07:14:39Z)
• RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-model GRPO [53.38929612273108]
  Causal autoregressive video diffusion models support real-time streaming generation by extrapolating future chunks from previously generated content.<n>We introduce the Real-time Autoregressive Video Extrapolation Network (RAVEN), a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints.
  arXiv  Detail & Related papers  (2026-05-14T17:59:30Z)
• Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity [57.83511884904928]
  We explore methods to accelerate training for autoregressive video generation models through empirical analyses.<n>Our results reveal that while training on fewer video frames significantly reduces training time, it also exacerbates error accumulation and introduces inconsistencies in the generated videos.<n>Inspired by Lipschitz continuity, we propose a Representation Continuity (ReCo) strategy to improve the consistency of generated videos.
  arXiv  Detail & Related papers  (2026-04-08T09:43:03Z)
• Astrolabe: Steering Forward-Process Reinforcement Learning for Distilled Autoregressive Video Models [58.3184497327891]
  Distilled autoregressive (AR) video models enable efficient streaming generation but frequently misalign with human visual preferences.<n>We present Astrolabe, an efficient online reinforcement learning framework tailored for distilled AR models.
  arXiv  Detail & Related papers  (2026-03-17T18:32:18Z)
• GDRO: Group-level Reward Post-training Suitable for Diffusion Models [55.948229011478304]
  Group-level rewards successfully align the model with the targeted reward.<n>Group-level Direct Reward Optimization (GDRO) is a new post-training paradigm for group-level reward alignment.<n>GDRO supports full offline training that saves the large time cost for image rollout sampling.<n>It is diffusion-sampler-independent, which eliminates the need for the ODE-to-SDE approximation to obtainity.
  arXiv  Detail & Related papers  (2026-01-05T11:47:18Z)
• Nested-ReFT: Efficient Reinforcement Learning for Large Language Model Fine-Tuning via Off-Policy Rollouts [25.205293698698867]
  We introduce Nested-ReFT, where a subset of layers of the target model acts as the behavior model to generate off-policy completions during training.<n>Our theoretical analysis shows that Nested-ReFT yields unbiased gradient estimates with controlled variance.<n>Our empirical analysis demonstrates improved computational efficiency measured as tokens/sec across multiple math reasoning benchmarks and model sizes.
  arXiv  Detail & Related papers  (2025-08-13T18:37:46Z)
• Improving Video Generation with Human Feedback [105.81833319891537]
  We develop a systematic pipeline that harnesses human feedback to mitigate video generation problems.<n>We introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy.
  arXiv  Detail & Related papers  (2025-01-23T18:55:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.