RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy

• URL: http://arxiv.org/abs/2607.27782v1
• Date: Thu, 30 Jul 2026 07:14:39 GMT
• Title: RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy
• Abstract summary: RedFlow is a fine-grained offline reinforcement learning framework for flow-matching VLA policies.<n>RedFlow redirects failure experiences into action-level corrective supervision for flow-matching VLA policies.<n>It consistently outperforms state-of-the-art offline RL baselines.
• Score: 24.413419742450913
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Flow-matching Vision-Language-Action (VLA) policies have shown strong potential for robotic manipulation but often suffer from compounding errors caused by distribution shifts during deployment. While offline reinforcement learning (RL) provides a practical way to improve deployed policies using rollout data, existing methods either ignore failure data or exploit it only at the trajectory level, resulting in low learning efficiency and persistent errors. We propose **RedFlow**, a fine-grained offline RL framework that redirects failure experiences into action-level corrective supervision for flow-matching VLA policies. RedFlow consists of two key components: (1) a **Context-Aware Corrective Matching** mechanism that identifies failure-inducing actions and retrieves successful alternatives from similar contexts as corrective targets, and (2) an **Adaptive Redirection Objective** that jointly reinforces successful actions, suppresses undesirable ones, and redirects recoverable failures toward corrective targets. By converting both successful and failed experiences into dense supervision, RedFlow enables robust recovery learning from mixed-quality data. Experiments on the LIBERO benchmark and three real-world manipulation tasks show that RedFlow consistently outperforms state-of-the-art offline RL baselines, improving the real-world success rate from 56.7% to 74.7%. It also matches strong on-policy methods (PPO, GRPO, and DDPO) while requiring roughly an order of magnitude fewer training samples.

Related papers

• Reversal Q-Learning [57.43956630070019]
  Reversal Q-learning is an off-policy RL algorithm that trains a flow policy based on prior data.<n>We show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.
  arXiv  Detail & Related papers  (2026-06-16T05:56:10Z)
• Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning [42.74142065376427]
  We propose Z-Perturbation Reinforcement Learning (ZPRL) to steer pretrained policies through a compact bottleneck latent.<n>In the real world, ZPRL improves the average success rate on four tasks by 33.7% over imitation base policies.
  arXiv  Detail & Related papers  (2026-05-19T14:43:26Z)
• CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL [56.969946199335716]
  We introduce a compositional constraint-based reward model for post-training embodied video generation models.<n>We propose CreFlow, a novel online RL framework with two key designs.
  arXiv  Detail & Related papers  (2026-05-14T02:18:58Z)
• RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models [90.39703013636868]
  RePO-VLA is a recovery-driven policy optimization framework.<n>It assigns distinct roles to success, recovery, and failure trajectories.<n>It raises adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.
  arXiv  Detail & Related papers  (2026-05-10T08:24:05Z)
• Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models [32.87104367896446]
  We propose an end-to-end framework that leverages failure trajectories as adaptive negative guidance for diffusion- and flow-based VLA policies.<n>AFIL uses a pretrained VLA to generate failure rollouts online, avoiding the need for handcrafted failure-mode design.<n>It then jointly trains Dual Action Generators (DAGs) for successful and failed behaviors while sharing a common vision-language backbone.
  arXiv  Detail & Related papers  (2026-05-08T19:57:11Z)
• Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning [10.037416068775853]
  We introduce Guided Flow Policy, which couples a multi-step flow-matching policy with a distilled one-step actor.<n>The actor directs the flow policy through weighted behavior cloning to focus on cloning high-value actions from the dataset.<n>This mutual guidance enables GFP to achieve state-of-the-art performance across 144 state and pixel-based tasks.
  arXiv  Detail & Related papers  (2025-12-03T17:05:58Z)
• One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow [56.13949180229929]
  We introduce a one-step generative policy for offline reinforcement learning that maps noise directly to actions via a residual reformulation of MeanFlow.<n>Our method achieves strong performance in both offline and offline-to-online reinforcement learning settings.
  arXiv  Detail & Related papers  (2025-11-17T06:34:17Z)
• Double Check My Desired Return: Transformer with Target Alignment for Offline Reinforcement Learning [64.6334337560557]
  Reinforcement learning via supervised learning (RvS) frames offline RL as a sequence modeling task.<n>Decision Transformer (DT) struggles to reliably align the actual achieved returns with specified target returns.<n>We propose Doctor, a novel approach that Double Checks the Transformer with target alignment for Offline RL.
  arXiv  Detail & Related papers  (2025-08-22T14:30:53Z)
• Boosting Offline Reinforcement Learning via Data Rebalancing [104.3767045977716]
  offline reinforcement learning (RL) is challenged by the distributional shift between learning policies and datasets. We propose a simple yet effective method to boost offline RL algorithms based on the observation that resampling a dataset keeps the distribution support unchanged. We dub our method ReD (Return-based Data Rebalance), which can be implemented with less than 10 lines of code change and adds negligible running time.
  arXiv  Detail & Related papers  (2022-10-17T16:34:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.