How to Learn from What a Human Would Avoid? Intervention-Aware World Models with Real-World RL for Dexterous Manipulation

• URL: http://arxiv.org/abs/2609.06009v1
• Date: Sat, 05 Sep 2026 10:28:23 GMT
• Title: How to Learn from What a Human Would Avoid? Intervention-Aware World Models with Real-World RL for Dexterous Manipulation
• Abstract summary: We present WHIRL, a safety-aware RL framework that transforms binary human interventions into forward-predictive signals for proactive risk avoidance.<n>We evaluate our framework on a 16-DoF LEAP Hand across tasks spanning convex and irregular object grasping, prismatic manipulation, and long-horizon multi-stage tasks.
• Score: 27.05205948648607
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-fingered dexterous manipulation remains a frontier for real-world reinforcement learning (RL) due to the high-dimensional action space and the prohibitive cost of hardware failures. While human-in-the-loop (HIL) RL allows operators to intervene before failures occur, current pipelines often treat these interventions as reactive corrections, discarding the rich safety signal inherent in the operator's decision to take control. In this paper, we ask: How can we learn from what a human would avoid? We present WHIRL, a safety-aware RL framework that transforms binary human interventions into forward-predictive signals for proactive risk avoidance. Our approach centers on an intervention-aware latent world model with four prediction heads: dynamics, reward, termination, and a novel per-state intervention-probability head that learns to predict the likelihood of a human takeover at future states. This head provides an actor-side risk-shaping term that discourages the policy from entering "intervention-prone" regions, modeling the operator's internal safety threshold. We evaluate our framework on a 16-DoF LEAP Hand across tasks spanning convex and irregular object grasping, prismatic manipulation, and long-horizon multi-stage tasks. Our results show that predictive risk-shaping enables the system to achieve a 96.7 percent success rate on complex grasping tasks while reducing the operator intervention burden by up to 84 percent in step-weighted terms. By closing the loop between human intuition and predictive world modeling, this work provides a practical safety-aware recipe for training complex dexterous agents in the real world while reducing operator fatigue and hardware-risk exposure.

Related papers

• GAINS: Leveraging Inconsistent Human Intervention Signals in Reinforcement Learning [35.461668686434244]
  We present a framework for leveraging inconsistent human intervention signals in reinforcement learning (RL)<n>We employ distributional RL with quantile Q-networks to model the return variability induced by sparse task rewards and inconsistent human interventions.<n>We evaluate GAINS on four diverse simulated manipulation tasks and two challenging real-world scenarios against state-of-the-art intervention-based methods.
  arXiv  Detail & Related papers  (2026-08-16T12:19:04Z)
• ROVE: Unlocking Human Interventions for Humanoid Manipulation via Reinforcement Learning [44.43333438148224]
  We propose ROVE, a reinforcement learning framework for humanoid VLA post-training with imperfect human interventions.<n>First, ROVE introduces a human-in-the-loop pipeline capable of collecting deployment and intervention data for humanoid manipulation.<n>Second, it utilizes Optimistic Value Estimation (OVE) to prioritize high-value behaviors from mixed-quality trajectories.
  arXiv  Detail & Related papers  (2026-06-15T17:45:06Z)
• OHP-RL: Online Human Preference as Guidance in Reinforcement Learning for Robot Manipulation [16.28822074948203]
  Online Human Preference as Guidance in Reinforcement Learning (OHP-RL) is a framework that leverages human interventions as preference information to guide policy learning.<n>OHP-RL consistently achieves strong success rates, faster convergence, and substantially lower human intervention effort than prior approaches.
  arXiv  Detail & Related papers  (2026-05-15T14:02:34Z)
• Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming [64.48633529149579]
  We propose a novel framework to expose the vulnerabilities of Vision-Language-Action (VLA) models against linguistic variations.<n>Our method reduces the average task success rate from 93.33% to 5.85%, demonstrating a scalable approach to stress-testing VLA agents.
  arXiv  Detail & Related papers  (2026-04-07T08:43:36Z)
• Safety, Security, and Cognitive Risks in World Models [0.0]
  World models are learned internal simulators of environment dynamics.<n>World models are rapidly becoming foundational to autonomous decision-making in robotics, autonomous vehicles, and agentic AI.<n>This paper surveys the world model landscape; introduces formal definitions of trajectory persistence and representational risk; presents a five-profile attacker taxonomy.<n>We provide an empirical proof-of-concept demonstrating trajectory-persistent adversarial attacks on a GRU-based RSSM.
  arXiv  Detail & Related papers  (2026-04-01T19:57:33Z)
• Risk-Aware World Model Predictive Control for Generalizable End-to-End Autonomous Driving [82.69496624372944]
  "Only driving like the expert" suffers from limited generalization.<n>Can an E2E-AD system make reliable decisions without any expert action supervision?<n>We propose a unified framework named Risk-aware World Model Predictive Control.
  arXiv  Detail & Related papers  (2026-02-26T17:32:30Z)
• Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Models [62.16655896700062]
  Activation steering is a technique to enhance the utility of Large Language Models (LLMs)<n>We show that it unintentionally introduces critical and under-explored safety risks.<n>Experiments reveal that these interventions act as a force multiplier, creating new vulnerabilities to jailbreaks and increasing attack success rates to over 80% on standard benchmarks.
  arXiv  Detail & Related papers  (2026-02-03T12:32:35Z)
• Real-world Reinforcement Learning from Suboptimal Interventions [39.23110010675281]
  SiLRI is a state-wise Lagrangian reinforcement learning algorithm for real-world robot manipulation tasks.<n>Built upon a human-as-copilot teleoperation system, our algorithm is evaluated through real-world experiments on diverse manipulation tasks.
  arXiv  Detail & Related papers  (2025-12-30T15:26:42Z)
• AD-R1: Closed-Loop Reinforcement Learning for End-to-End Autonomous Driving with Impartial World Models [75.214287449744]
  We introduce a framework for post-training policy refinement built around an Impartial World Model.<n>Our primary contribution is to teach this model to be honest about danger.<n>We demonstrate through extensive experiments, that our model significantly outperforms baselines in predicting failures.
  arXiv  Detail & Related papers  (2025-11-25T13:57:24Z)
• BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Models [57.5404308854535]
  Safety backdoor attacks in large language models (LLMs) enable the stealthy triggering of unsafe behaviors while evading detection during normal interactions. We present BEEAR, a mitigation approach leveraging the insight that backdoor triggers induce relatively uniform drifts in the model's embedding space. Our bi-level optimization method identifies universal embedding perturbations that elicit unwanted behaviors and adjusts the model parameters to reinforce safe behaviors against these perturbations.
  arXiv  Detail & Related papers  (2024-06-24T19:29:47Z)
• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis [63.532413807686524]
  This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL) We propose a new architecture that handles the trade-off between efficient progress and safety during exploration.
  arXiv  Detail & Related papers  (2023-12-18T16:09:43Z)
• Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization [38.21629972247463]
  We develop a novel human-in-the-loop learning method called Human-AI Copilot Optimization (HACO) The proposed HACO effectively utilizes the data both from the trial-and-error exploration and human's partial demonstration to train a high-performing agent. experiments show that HACO achieves a substantially high sample efficiency in the safe driving benchmark.
  arXiv  Detail & Related papers  (2022-02-17T06:29:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.