ORPA: Online Residual Policy Adaptation for Robot Manipulation Control with Human Feedback

• URL: http://arxiv.org/abs/2608.17323v1
• Date: Tue, 18 Aug 2026 03:30:12 GMT
• Title: ORPA: Online Residual Policy Adaptation for Robot Manipulation Control with Human Feedback
• Abstract summary: Online Residual Policy Adaptation (ORPA) is a framework that enables feedback-driven correction of robot actions without modifying the underlying policy parameters.<n>ORPA augments a pretrained control policy with a lightweight, feedback-conditioned module that predicts residual adjustments directly in joint space.<n>We evaluate ORPA on a set of precision-sensitive manipulation tasks using the ALOHA platform, demonstrating improvements in success rate and recovery from small perturbations compared to baseline control policies and rule-based inverse kinematics corrections.
• Score: 1.711436622393758
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robotic manipulation policies trained via imitation learning, such as Action Chunking with Transformers (ACT), can achieve strong performance under ideal conditions but often remain sensitive to small execution errors and distribution shifts. Correcting these failures typically requires dataset aggregation and full-policy retraining, which is computationally expensive and unsuitable for real-time deployment. In this work, we propose Online Residual Policy Adaptation (ORPA), a framework that enables immediate, feedback-driven correction of robot actions without modifying the underlying policy parameters. ORPA augments a pretrained control policy with a lightweight, feedback-conditioned module that predicts residual adjustments directly in joint space, allowing the system to adapt its behavior at runtime. We evaluate ORPA on a set of precision-sensitive manipulation tasks using the ALOHA platform, demonstrating improvements in success rate and recovery from small perturbations compared to baseline control policies and rule-based inverse kinematics corrections.

Related papers

• Adapting Generalist Robot Policies with Semantic Reinforcement Learning [59.009122725071386]
  Generalist robot policies learn a diverse repertoire of behaviors from large-scale pretraining.<n>We propose Semantic Action Reinforcement Learning (SARL) to learn to optimize this prompt space through online interaction.<n>We show SARL unlocks fundamentally new capabilities -- adapting VLA behavior to solve complex, long-horizon tasks.
  arXiv  Detail & Related papers  (2026-06-30T17:00:33Z)
• Improving Robotic Generalist Policies via Flow Reversal Steering [92.97477771370428]
  Generalist policies can learn a wide range of skills from diverse robot datasets.<n>In order to solve or improve on challenging new tasks, we need a way to infer and invoke the appropriate actions from the policy's rich behavioral prior.<n>We propose Flow Reversal Steering (FRS): a method that takes suboptimal but reasonable'' actions, finds their latent noises by passing them through the flow policy in reverse, and maps them to nearby generalist action modes.
  arXiv  Detail & Related papers  (2026-06-11T17:59:45Z)
• Beyond Action Residuals: Real-World Robot Policy Steering via Bottleneck Latent Reinforcement Learning [42.74142065376427]
  We propose Z-Perturbation Reinforcement Learning (ZPRL) to steer pretrained policies through a compact bottleneck latent.<n>In the real world, ZPRL improves the average success rate on four tasks by 33.7% over imitation base policies.
  arXiv  Detail & Related papers  (2026-05-19T14:43:26Z)
• OGPO: Sample Efficient Full-Finetuning of Generative Control Policies [53.42266064673132]
  Generative control policies (GCPs) have emerged as effective parameterizations for robot learning.<n>This work introduces Off-policy Generative Policy Optimization (OGPO), a sample-efficient for finetuning GCPs.<n>OGPO achieves state-of-the-art performance on manipulation tasks spanning multi-task settings, high-precision insertion, and dexterous control.
  arXiv  Detail & Related papers  (2026-05-04T18:36:40Z)
• Hypernetwork-Conditioned Reinforcement Learning for Robust Control of Fixed-Wing Aircraft under Actuator Failures [0.8164433158925594]
  This paper presents a reinforcement learning-based path-following controller for a small uncrewed aircraft system.<n>We consider parameter-efficient formulations based on Feature-wise Linear Modulation (FiLM) and Low-Rank Adaptation (LoRA)<n>We demonstrate that hypernetwork-conditioned policies can improve robustness compared to standard multilayer perceptron policies.
  arXiv  Detail & Related papers  (2026-04-03T18:50:31Z)
• FlowCorrect: Efficient Interactive Correction of Generative Flow Policies for Robotic Manipulation [0.7666240799116112]
  FlowCorrect is a modular interactive imitation learning approach that enables deployment-time adaptation of flow-matching manipulation policies.<n>We evaluate on a real-world robot across four tabletop tasks: pick-and-place, pouring, cup uprighting, and insertion.<n>With a low correction budget, FlowCorrect achieves an 80% success rate on previously failed cases.
  arXiv  Detail & Related papers  (2026-02-25T16:06:49Z)
• RFS: Reinforcement Learning with Residual Flow Steering for Dexterous Manipulation [7.500999283386335]
  Residual Flow Steering (RFS) is a data-efficient reinforcement learning framework for adapting pretrained generative policies.<n>RFS steers a pretrained flow-matching policy by jointly optimizing a residual action and a latent noise distribution.<n>We demonstrate the effectiveness of RFS on dexterous manipulation tasks, showing efficient fine-tuning in both simulation and real-world settings.
  arXiv  Detail & Related papers  (2026-02-02T08:11:57Z)
• Double Check My Desired Return: Transformer with Target Alignment for Offline Reinforcement Learning [64.6334337560557]
  Reinforcement learning via supervised learning (RvS) frames offline RL as a sequence modeling task.<n>Decision Transformer (DT) struggles to reliably align the actual achieved returns with specified target returns.<n>We propose Doctor, a novel approach that Double Checks the Transformer with target alignment for Offline RL.
  arXiv  Detail & Related papers  (2025-08-22T14:30:53Z)
• FDPP: Fine-tune Diffusion Policy with Human Preference [57.44575105114056]
  Fine-tuning Diffusion Policy with Human Preference learns a reward function through preference-based learning.<n>This reward is then used to fine-tune the pre-trained policy with reinforcement learning.<n>Experiments demonstrate that FDPP effectively customizes policy behavior without compromising performance.
  arXiv  Detail & Related papers  (2025-01-14T17:15:27Z)
• Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization [63.75188254377202]
  Deep reinforcement learning algorithms can perform poorly in real-world tasks due to discrepancy between source and target environments. We propose a novel model-free actor-critic algorithm to learn robust policies without modeling the disturbance in advance. Experiments in several robot control tasks demonstrate that SCPO learns robust policies against the disturbance in transition dynamics.
  arXiv  Detail & Related papers  (2021-12-20T13:13:05Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.