Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning

• URL: http://arxiv.org/abs/2608.15088v1
• Date: Sat, 15 Aug 2026 07:13:59 GMT
• Title: Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning
• Abstract summary: Human-in-the-loop (HIL) online reinforcement learning for real robots must absorb human interventions quickly while continuing to improve beyond the human prior.<n>We present a training method for this setting based on two components.<n>emphMC Q-chunk critic regresses chunk-level action values onto Monte Carlo returns from a replay buffer.<n>emphmax-Q selective imitation updates the actor by imitating, at each state, the higher-$Q$ action between the current policy action and a buffer sample.
• Score: 1.96243636752331
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human-in-the-loop (HIL) online reinforcement learning for real robots must absorb human interventions quickly while continuing to improve beyond the human prior. We present a training method for this setting based on two components. First, an \emph{MC Q-chunk} critic regresses chunk-level action values onto Monte Carlo returns from the replay buffer, performing sample-average (behavior) policy evaluation so that intervention trajectories are credited directly rather than diluted by current-policy TD backups. Second, \emph{max-Q selective imitation} updates the actor by imitating, at each state, the higher-$Q$ action between the current policy action and a buffer sample under a hard winner-take-all rule. This rule automatically switches between learning from interventions and on-policy self-improvement: when the autonomous policy is stronger, targets align with the policy distribution, reducing the policy--target-sample gap that otherwise induces execution-time distribution shift. In practice we score candidates with a standard critic ensemble mean to reduce comparison noise, without softening targets or introducing score-gap thresholds. On a real USB pick-and-insertion task with 20 demonstrations, ACT QChunk-MCBC attains 99\% success within 30 minutes of HIL training, whereas HIL-SERL requires about 5 hours to converge. In simulation on Peg Insertion and Square, ACT/Flow Q-chunk variants similarly reach $\ge$96\% success within roughly half an hour of effective training, outperforming HIL-SERL, EXPO, and E2HiL on the success--time frontier.

Related papers

• PAVE: Predictive Alignment and Value-Guided Evolution for World-Action Policies [17.87859739145508]
  Direct vision-language-action policies generate continuous robot actions efficiently, but standard behavior cloning leaves two complementary gaps.<n>We introduce method, a direct world-action policy that combines outcome-agnostic predictive learning with outcome-aware policy improvement.
  arXiv  Detail & Related papers  (2026-08-31T07:36:11Z)
• Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition [16.480150894507908]
  We introduce a unified framework that combines centralized training with decentralized execution (CTDE) and a Hybrid Reward Architecture (HRA)<n> Experimental results demonstrate that the proposed framework substantially improves both sample efficiency and policy performance.<n>Compared with a state-of-the-art baseline, our method improves the success rate from 60% to 80% on tennis ball pick-and-place.
  arXiv  Detail & Related papers  (2026-08-10T15:54:25Z)
• VIMPO: Value-Implicit Policy Optimization for LLMs [106.88933849641272]
  Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token.<n>Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability.<n>We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from optimality conditions of KL-regularized reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-18T09:44:12Z)
• QPILOTS: Efficient Test-Time Q-Steering for Flow Policies [20.217020870532686]
  QPILOTS is a method that leaves the original policy unmodified and steers the denoising process at inference time.<n>On a standard offline-to-online RL benchmark, QPILOTS achieves the best aggregate performance, reaching an average success rate of 90% across 50 tasks.
  arXiv  Detail & Related papers  (2026-06-11T18:22:03Z)
• Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning [50.738952715864116]
  Expressive continuous control policies form the backbone of advances in scaling imitation learning for simulated and real robot control.<n>We propose QGF (Q-Guided Flow), an RL algorithm that performs policy optimization entirely at test time.<n> Empirically, QGF outperforms prior test-time RL methods on single-task and goal-conditioned offline RL benchmarks.
  arXiv  Detail & Related papers  (2026-06-09T16:45:57Z)
• Potential-Guided Flow Matching for Vision-Language-Action Policy Improvement [27.695600755960736]
  We introduce ForesightFlow, a self-guided flow-matching policy.<n>It augments each generated action chunk with a learned success-potential trajectory.<n>It scores candidate actions, enabling best-of-$K$ inference without an external critic.
  arXiv  Detail & Related papers  (2026-06-03T14:49:35Z)
• Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation [40.17737666526493]
  We propose PACT, a Preference-calibrated Actor-Critic Training framework.<n>We first design a progress model that learns from human demonstration and identifies suboptimal segments for credit correction.<n>We then build preference pairs to define a counterfactual advantage that penalizes Bellman targets of the identified suboptimal segment.
  arXiv  Detail & Related papers  (2026-06-02T17:38:25Z)
• Learn Where Outcomes Diverge: Efficient VLA RL via Probabilistic Chunk Masking [5.238545250784642]
  This paper presents Probabilistic Chunk Masking (PCM), a drop-in modification to GRPO that allocates gradient to a small, probabilistically selected subset of chunks per trajectory.<n>On three LIBERO benchmarks, PCM matches the final success rate of standard GRPO while achieving 2.38 times wall-clock speedup, 4.8 times faster gradient updates, and 60% lower peak activation memory.
  arXiv  Detail & Related papers  (2026-05-15T16:33:59Z)
• One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow [56.13949180229929]
  We introduce a one-step generative policy for offline reinforcement learning that maps noise directly to actions via a residual reformulation of MeanFlow.<n>Our method achieves strong performance in both offline and offline-to-online reinforcement learning settings.
  arXiv  Detail & Related papers  (2025-11-17T06:34:17Z)
• Fast Adaptation with Behavioral Foundation Models [82.34700481726951]
  Unsupervised zero-shot reinforcement learning has emerged as a powerful paradigm for pretraining behavioral foundation models.<n>Despite promising results, zero-shot policies are often suboptimal due to errors induced by the unsupervised training process.<n>We propose fast adaptation strategies that search in the low-dimensional task-embedding space of the pre-trained BFM to rapidly improve the performance of its zero-shot policies.
  arXiv  Detail & Related papers  (2025-04-10T16:14:17Z)
• One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation [80.71541671907426]
  OneStep Diffusion Policy (OneDP) is a novel approach that distills knowledge from pre-trained diffusion policies into a single-step action generator. OneDP significantly accelerates response times for robotic control tasks.
  arXiv  Detail & Related papers  (2024-10-28T17:54:31Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.