RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulation

• URL: http://arxiv.org/abs/2602.01789v3
• Date: Thu, 05 Feb 2026 09:22:39 GMT
• Title: RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulation
• Abstract summary: Residual Flow Steering (RFS) is a data-efficient reinforcement learning framework for adapting pretrained generative policies.<n>RFS steers a pretrained flow-matching policy by jointly optimizing a residual action and a latent noise distribution.<n>We demonstrate the effectiveness of RFS on dexterous manipulation tasks, showing efficient fine-tuning in both simulation and real-world settings.
• Score: 7.500999283386335
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Imitation learning has emerged as an effective approach for bootstrapping sequential decision-making in robotics, achieving strong performance even in high-dimensional dexterous manipulation tasks. Recent behavior cloning methods further leverage expressive generative models, such as diffusion models and flow matching, to represent multimodal action distributions. However, policies pretrained in this manner often exhibit limited generalization and require additional fine-tuning to achieve robust performance at deployment time. Such adaptation must preserve the global exploration benefits of pretraining while enabling rapid correction of local execution errors. We propose Residual Flow Steering(RFS), a data-efficient reinforcement learning framework for adapting pretrained generative policies. RFS steers a pretrained flow-matching policy by jointly optimizing a residual action and a latent noise distribution, enabling complementary forms of exploration: local refinement through residual corrections and global exploration through latent-space modulation. This design allows efficient adaptation while retaining the expressive structure of the pretrained policy. We demonstrate the effectiveness of RFS on dexterous manipulation tasks, showing efficient fine-tuning in both simulation and real-world settings when adapting pretrained base policies. Project website:https://weirdlabuw.github.io/rfs.

Related papers

• IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning [57.16513298507272]
  We propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR)<n>IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization.<n>For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional.
  arXiv  Detail & Related papers  (2026-08-11T08:20:02Z)
• FlowDAgger: Human-in-the-Loop Adaptation of Generative Robot Policies in Latent Space [15.351960740594201]
  FlowDAgger is a sample- and compute-efficient method for adapting frozen generative robot policies from human interventions in latent space.<n>Our key idea is action inversion: each human expert action is mapped to the noise that would have produced it under the frozen base policy.<n>We evaluate FlowDAgger in simulation and on real-world bimanual and single-arm manipulation.
  arXiv  Detail & Related papers  (2026-07-09T19:07:33Z)
• ADORN: Adaptive Drift handling for Open RAN using Reinforcement Learning [0.9577207941409713]
  Dynamic traffic variations in Open Radio Access Networks (O-RAN) lead to drift.<n>Traditional retraining approaches maintain forecasting accuracy but incur high computational cost.<n>This work proposes a Q-learning-based adaptive retraining approach that formulates the retraining decision as a Markov Decision Process.
  arXiv  Detail & Related papers  (2026-07-09T13:05:42Z)
• Lagrangian Perturbation Diffusion Steering: Latent Reinforcement Learning for Generative Policies [5.170807667319543]
  We propose a lightweight adaptation method that improves a frozen generative policy by learning a compact noise-space perturbation before decoding.<n>Across RoboMimic manipulation, OpenAI Gym locomotion, and Adroit dexterous manipulation benchmarks, LP-DS improves sample efficiency, success, and return while maintaining higher action-space entropy than unconstrained noise-space steering.
  arXiv  Detail & Related papers  (2026-05-31T10:40:28Z)
• Implicit Drifting Policy: One-Step Action Generation via Conditional Expert Geometry [46.0185525503119]
  Implicit Drifting Policy (IDP) is a one-step imitation learning framework for robot control.<n>IDP brings the training-time correction of Drifting into policy learning without explicit vector field estimation.
  arXiv  Detail & Related papers  (2026-05-31T08:39:57Z)
• AnchorRefine: Synergy-Manipulation Based on Trajectory Anchor and Residual Refinement for Vision-Language-Action Models [60.04879435087352]
  Vision-language-action (VLA) policies generate actions within a single unified space.<n>We propose AnchorRefine, a hierarchical framework that factorizes VLA action modeling into trajectory anchor and residual refinement.<n> Experiments on LIBERO, CALVIN, and real-robot tasks demonstrate that AnchorRefine consistently improves both regression-based and diffusion-based VLA backbones.
  arXiv  Detail & Related papers  (2026-04-20T04:25:24Z)
• Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuning [34.29473529235131]
  LifeLong-RFT is a simple yet effective Reinforcement Fine-Tuning (RFT) strategy for VLA models.<n>LifeLong-RFT integrates chunking-level on-policy reinforcement learning with the proposed Multi-Dimensional Process Reward (MDPR) mechanism.<n>Our method achieves a 22% gain in average success rate over SFT, while effectively adapting to new tasks using only 20% of the training data.
  arXiv  Detail & Related papers  (2026-02-11T04:05:03Z)
• DFPO: Scaling Value Modeling via Distributional Flow towards Robust and Generalizable LLM Post-Training [94.568675548967]
  Training reinforcement learning (RL) systems in real-world environments remains challenging due to noisy supervision and poor out-of-domain generalization.<n>Recent distributional RL methods improve robustness by modeling values with multiple quantile points, but they still learn each quantile independently as a scalar.<n>We propose DFPO, a robust distributional RL framework that models values as continuous flows across time steps.
  arXiv  Detail & Related papers  (2026-02-05T17:07:42Z)
• Towards Bridging the Gap between Large-Scale Pretraining and Efficient Finetuning for Humanoid Control [18.618486126491334]
  We find that off-policy Soft Actor-Critic (SAC) reliably supports large-scale pretraining of humanoid locomotion policies.<n>SAC-pretrained policies can be finetuned in new environments and out-of-distribution tasks using model-based methods.<n>Overall, the approach couples the wall-clock efficiency of large-scale simulation during pretraining with the sample efficiency of model-based learning during fine-tuning.
  arXiv  Detail & Related papers  (2026-01-29T07:43:24Z)
• Iterative Refinement of Flow Policies in Probability Space for Online Reinforcement Learning [56.47948583452555]
  We introduce the Stepwise Flow Policy (SWFP) framework, founded on the key insight that discretizing the flow matching inference process via a fixed-step Euler scheme aligns it with the variational Jordan-Kinderlehrer-Otto principle from optimal transport.<n>SWFP decomposes the global flow into a sequence of small, incremental transformations between proximate distributions.<n>This decomposition yields an efficient algorithm that fine-tunes pre-trained flows via a cascade of small flow blocks, offering significant advantages.
  arXiv  Detail & Related papers  (2025-10-17T07:43:51Z)
• Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidance [63.33213516925946]
  We introduce textbfAlign-Then-stEer (textttATE), a novel, data-efficient, and plug-and-play adaptation framework.<n>Our work presents a general and lightweight solution that greatly enhances the practicality of deploying VLA models to new robotic platforms and tasks.
  arXiv  Detail & Related papers  (2025-09-02T07:51:59Z)
• Fast and Stable Diffusion Planning through Variational Adaptive Weighting [3.745003761050674]
  Diffusion models have recently shown promise in offline RL.<n>These methods often suffer from high training costs and slow convergence.<n>We introduce a closed-form approximation method for its online estimation under the flow-based generative modeling framework.<n> Experimental results on Maze2D and Kitchen tasks show that our method achieves competitive performance with up to 10 times fewer training steps.
  arXiv  Detail & Related papers  (2025-06-20T02:12:04Z)
• A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning [45.19254609437857]
  Online reinforcement learning (RL) excels in complex, safety-critical domains but suffers from sample inefficiency, training instability, and limited interpretability.<n>Data attribution provides a principled way to trace model behavior back to training samples.<n>We propose an algorithm, iterative influence-based filtering (IIF), for online RL training that iteratively performs experience filtering to refine policy updates.
  arXiv  Detail & Related papers  (2025-05-25T19:25:57Z)
• Fast Adaptation with Behavioral Foundation Models [82.34700481726951]
  Unsupervised zero-shot reinforcement learning has emerged as a powerful paradigm for pretraining behavioral foundation models.<n>Despite promising results, zero-shot policies are often suboptimal due to errors induced by the unsupervised training process.<n>We propose fast adaptation strategies that search in the low-dimensional task-embedding space of the pre-trained BFM to rapidly improve the performance of its zero-shot policies.
  arXiv  Detail & Related papers  (2025-04-10T16:14:17Z)
• A Regularized Implicit Policy for Offline Reinforcement Learning [54.7427227775581]
  offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. We propose a framework that supports learning a flexible yet well-regularized fully-implicit policy. Experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs.
  arXiv  Detail & Related papers  (2022-02-19T20:22:04Z)
• Deep Reinforcement Learning with Robust and Smooth Policy [90.78795857181727]
  We propose to learn a smooth policy that behaves smoothly with respect to states. We develop a new framework -- textbfSmooth textbfRegularized textbfReinforcement textbfLearning ($textbfSR2textbfL$), where the policy is trained with smoothness-inducing regularization. Such regularization effectively constrains the search space, and enforces smoothness in the learned policy.
  arXiv  Detail & Related papers  (2020-03-21T00:10:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.