Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation

• URL: http://arxiv.org/abs/2606.03949v1
• Date: Tue, 02 Jun 2026 17:38:25 GMT
• Title: Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation
• Abstract summary: We propose PACT, a Preference-calibrated Actor-Critic Training framework.<n>We first design a progress model that learns from human demonstration and identifies suboptimal segments for credit correction.<n>We then build preference pairs to define a counterfactual advantage that penalizes Bellman targets of the identified suboptimal segment.
• Score: 40.17737666526493
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human-in-the-loop reinforcement learning (HIL-RL) improves sample efficiency in real-robot manipulation through online human intervention. However, successful trajectories may include suboptimal actions that deviate from the desired task-execution path and force human intervention. Existing HIL-RL methods typically apply the consistent credit assignment principle to all transitions, uniformly propagating discounted terminal rewards through suboptimal segments, ignoring the actual contribution of each transition to task success. This overestimates Q-values for critic learning and indirectly misguides actor updates toward suboptimal behavior patterns. To this end, we propose PACT, a Preference-calibrated Actor-Critic Training framework that leverages the implicit preference signals induced by intervention to perform credit reassignment on identified suboptimal segments while directly guiding policy training for unbiased critic-actor learning. Specifically, we first design a progress model that learns from human demonstration and identifies suboptimal segments for credit correction. Then, from the human action and resampled policy action at the intervention state, we build preference pairs to define a counterfactual advantage that penalizes Bellman targets of the identified suboptimal segment, enabling directional credit calibration. Moreover, we directly align the policy with human corrective actions in the bounded mean space, providing an additional signal beyond critic-guided updates. Across five real-robot manipulation tasks, PACT improves the average success rate by 24.5% and achieves 1.3 times faster convergence, thereby improving both RL sample efficiency and performance. Code is available at https://anonymous.4open.science/r/HILRL-A1X-BC05.

Related papers

• Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning [1.96243636752331]
  Human-in-the-loop (HIL) online reinforcement learning for real robots must absorb human interventions quickly while continuing to improve beyond the human prior.<n>We present a training method for this setting based on two components.<n>emphMC Q-chunk critic regresses chunk-level action values onto Monte Carlo returns from a replay buffer.<n>emphmax-Q selective imitation updates the actor by imitating, at each state, the higher-$Q$ action between the current policy action and a buffer sample.
  arXiv  Detail & Related papers  (2026-08-15T07:13:59Z)
• IADD-TR: Intervention-Aware Dynamics Decoupling with Targeted Regularization for Model-Based Reinforcement Learning [57.16513298507272]
  We propose IADD-TR, a unified framework combining Intervention-Aware Dynamics Decoupling (IADD) and Targeted Regularization (TR)<n>IADD factorizes transitions into an action-intervention stage and an action-free natural evolution stage, using a zero-action anchor to resolve the non-uniqueness of this two-stage factorization for robust generalization.<n>For policy learning, we derive TR from the efficient influence function of a replay-state policy-gradient functional.
  arXiv  Detail & Related papers  (2026-08-11T08:20:02Z)
• Potential-Guided Flow Matching for Vision-Language-Action Policy Improvement [27.695600755960736]
  We introduce ForesightFlow, a self-guided flow-matching policy.<n>It augments each generated action chunk with a learned success-potential trajectory.<n>It scores candidate actions, enabling best-of-$K$ inference without an external critic.
  arXiv  Detail & Related papers  (2026-06-03T14:49:35Z)
• LoopVLA: Learning Sufficiency in Recurrent Refinement for Vision-Language-Action Models [13.30873593845724]
  LoopVLA is a recurrent Vision-Language-Action architecture that learns representation refinement, action prediction, and sufficiency estimation.<n>We show that LoopVLA pushes the efficiency-performance frontier of VLA policies, reducing parameters by 45% and improving inference throughput by up to 1.7 times.
  arXiv  Detail & Related papers  (2026-05-11T03:51:22Z)
• Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learning [22.17044827069627]
  We propose a plug-and-play training sample replacer to replace low-value actions with high-value actions generated by a stable actor.<n>Experiments show that PAR consistently improves performance and approaches state-of-the-art when combined with the basic TD3+BC.
  arXiv  Detail & Related papers  (2026-02-07T08:44:27Z)
• Human-in-the-loop Online Rejection Sampling for Robotic Manipulation [55.99788088622936]
  Hi-ORS stabilizes value estimation by filtering out negatively rewarded samples during online fine-tuning.<n>Hi-ORS fine-tunes a pi-base policy to master contact-rich manipulation in just 1.5 hours of real-world training.
  arXiv  Detail & Related papers  (2025-10-30T11:53:08Z)
• Learning to Reason as Action Abstractions with Scalable Mid-Training RL [55.24192942739207]
  An effective mid-training phase should identify a compact set of useful actions and enable fast selection.<n>We propose Reasoning as Action Abstractions (RA3), a scalable mid-training algorithm.
  arXiv  Detail & Related papers  (2025-09-30T05:34:20Z)
• Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning [47.57615889991631]
  For continuous action spaces, actor-critic methods are widely used in online reinforcement learning (RL)<n>This study examines the effectiveness of incorporating the Bellman optimality operator into actor-critic frameworks.
  arXiv  Detail & Related papers  (2025-06-06T10:46:20Z)
• Fast Adaptation with Behavioral Foundation Models [82.34700481726951]
  Unsupervised zero-shot reinforcement learning has emerged as a powerful paradigm for pretraining behavioral foundation models.<n>Despite promising results, zero-shot policies are often suboptimal due to errors induced by the unsupervised training process.<n>We propose fast adaptation strategies that search in the low-dimensional task-embedding space of the pre-trained BFM to rapidly improve the performance of its zero-shot policies.
  arXiv  Detail & Related papers  (2025-04-10T16:14:17Z)
• Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewards [38.056359612828466]
  We propose a novel value-based deep RL algorithm called Iterative learning from Corrective actions and Proxy rewards (ICoPro) We experimentally validate our proposition on a variety of tasks (Atari games and autonomous driving on highway)
  arXiv  Detail & Related papers  (2024-10-08T08:04:09Z)
• REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
  A misalignment between the reward function and human preferences can lead to catastrophic outcomes in the real world.<n>Recent methods aim to mitigate misalignment by learning reward functions from human preferences.<n>We propose a novel concept of reward regularization within the robotic RLHF framework.
  arXiv  Detail & Related papers  (2023-12-22T04:56:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.