Improving Robotic Generalist Policies via Flow Reversal Steering

• URL: http://arxiv.org/abs/2606.13675v2
• Date: Fri, 12 Jun 2026 16:52:13 GMT
• Title: Improving Robotic Generalist Policies via Flow Reversal Steering
• Abstract summary: Generalist policies can learn a wide range of skills from diverse robot datasets.<n>In order to solve or improve on challenging new tasks, we need a way to infer and invoke the appropriate actions from the policy's rich behavioral prior.<n>We propose Flow Reversal Steering (FRS): a method that takes suboptimal but reasonable'' actions, finds their latent noises by passing them through the flow policy in reverse, and maps them to nearby generalist action modes.
• Score: 92.97477771370428
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generalist policies can learn a wide range of skills from diverse robot datasets. In order to solve or improve on challenging new tasks, we need a way to infer and invoke the appropriate actions from the policy's rich behavioral prior, especially when directly commanding the policy fails. We focus on flow matching generalists and propose Flow Reversal Steering (FRS): a method that takes suboptimal but ``reasonable'' actions, finds their latent noises by passing them through the flow policy in reverse, and maps them to nearby generalist action modes. We evaluate FRS across many simulated and real-world manipulation settings. First, FRS can turn coarse semantic guidance from humans or vision-language models (VLMs) into corresponding good robot actions, improving zero-shot control. These gains can be distilled with behavioral cloning by training an auxiliary policy to output noises that the generalist maps to good actions -- showing up to 95% absolute task success rate boosts in under a minute of training. Finally, FRS enables policy improvement by bootstrapping reinforcement learning with semantic knowledge, improving on several tasks that standard RL fails to improve on.

Related papers

• RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation [36.16163751472016]
  offline reinforcement learning framework makes AR-NF policies practical for robotic manipulation.<n>RoMAN-Flow employs a sampling-free, advantage-weighted likelihood objective.<n>Experiments across simulated manipulation benchmarks and real-world robotic platforms demonstrate that RoMAN-Flow achieves competitive policy performance.
  arXiv  Detail & Related papers  (2026-08-20T16:07:56Z)
• Adapting Generalist Robot Policies with Semantic Reinforcement Learning [59.009122725071386]
  Generalist robot policies learn a diverse repertoire of behaviors from large-scale pretraining.<n>We propose Semantic Action Reinforcement Learning (SARL) to learn to optimize this prompt space through online interaction.<n>We show SARL unlocks fundamentally new capabilities -- adapting VLA behavior to solve complex, long-horizon tasks.
  arXiv  Detail & Related papers  (2026-06-30T17:00:33Z)
• Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning [42.74142065376427]
  We propose Z-Perturbation Reinforcement Learning (ZPRL) to steer pretrained policies through a compact bottleneck latent.<n>In the real world, ZPRL improves the average success rate on four tasks by 33.7% over imitation base policies.
  arXiv  Detail & Related papers  (2026-05-19T14:43:26Z)
• Flow Policy Gradients for Robot Control [67.61978635211048]
  Flow matching policy gradients can be made effective for training and fine-tuning more expressive policies.<n>We show how policies can exploit the flow representation for exploration when training from scratch, as well as improved fine-tuning robustness over baselines.
  arXiv  Detail & Related papers  (2026-02-02T18:56:49Z)
• Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning [10.037416068775853]
  We introduce Guided Flow Policy, which couples a multi-step flow-matching policy with a distilled one-step actor.<n>The actor directs the flow policy through weighted behavior cloning to focus on cloning high-value actions from the dataset.<n>This mutual guidance enables GFP to achieve state-of-the-art performance across 144 state and pixel-based tasks.
  arXiv  Detail & Related papers  (2025-12-03T17:05:58Z)
• Reinforcement Learning for Flow-Matching Policies [9.308313682356285]
  Flow-matching policies have emerged as a powerful paradigm for generalist robotics.<n>This work explores training flow-matching policies via reinforcement learning to surpass the original demonstration policy performance.
  arXiv  Detail & Related papers  (2025-07-20T18:15:18Z)
• Fast Adaptation with Behavioral Foundation Models [82.34700481726951]
  Unsupervised zero-shot reinforcement learning has emerged as a powerful paradigm for pretraining behavioral foundation models.<n>Despite promising results, zero-shot policies are often suboptimal due to errors induced by the unsupervised training process.<n>We propose fast adaptation strategies that search in the low-dimensional task-embedding space of the pre-trained BFM to rapidly improve the performance of its zero-shot policies.
  arXiv  Detail & Related papers  (2025-04-10T16:14:17Z)
• FDPP: Fine-tune Diffusion Policy with Human Preference [57.44575105114056]
  Fine-tuning Diffusion Policy with Human Preference learns a reward function through preference-based learning.<n>This reward is then used to fine-tune the pre-trained policy with reinforcement learning.<n>Experiments demonstrate that FDPP effectively customizes policy behavior without compromising performance.
  arXiv  Detail & Related papers  (2025-01-14T17:15:27Z)
• FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning [74.25049012472502]
  FLaRe is a large-scale Reinforcement Learning framework that integrates robust pre-trained representations, large-scale training, and gradient stabilization techniques. Our method aligns pre-trained policies towards task completion, achieving state-of-the-art (SoTA) performance on previously demonstrated and on entirely novel tasks and embodiments.
  arXiv  Detail & Related papers  (2024-09-25T03:15:17Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.