DexPIE: Stable Dexterous Policy Improvement from Real-World Experience

• URL: http://arxiv.org/abs/2606.09615v1
• Date: Mon, 08 Jun 2026 15:21:45 GMT
• Title: DexPIE: Stable Dexterous Policy Improvement from Real-World Experience
• Abstract summary: Dexterous manipulation presents substantial challenges for imitation learning due to its high-dimensional action space and complex dynamics.<n>We propose DexPIE, a post-training framework for dexterous policy improvement from experience collected through real-world deployment.<n>Across three challenging real-world dexterous manipulation tasks, DexPIE achieves a 37% improvement in success rate over the demonstration-based reference policy.
• Score: 33.644712897849196
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dexterous manipulation presents substantial challenges for imitation learning due to its high-dimensional action space and complex contact-rich dynamics. Policies trained purely from demonstrations often suffer from compounding errors during deployment and require large amounts of expert data to achieve reliable performance. To move beyond the limitations of demonstration data, in this work, we propose DexPIE, a post-training framework for dexterous policy improvement from experience collected through real-world deployment. First, DexPIE enables effective exploration coverage through a dexterous-hand-adapted intervention system and multi-stage DAgger-style data collection across initial and intermediate task stages, providing reliable supervision for accurate policy evaluation. To reduce temporal noise between post-training rollouts and demonstration data, we introduce asynchronous inference in the relative action space, which better aligns rollout data with demonstrated behavior and allows the critic to learn a value function induced by a more consistent underlying policy. Finally, DexPIE improves the policy through conditioning on a continuous optimality indicator, allowing the policy to leverage the quality of data in a more fine-grained manner. Across three challenging real-world dexterous manipulation tasks, DexPIE achieves a 37% improvement in success rate over the demonstration-based reference policy, outperforming all baseline methods and demonstrating stronger robustness. The source code and dataset will be made publicly available.

Related papers

• Do Robotic World Models Really Follow Actions? Diagnosing and Aligning Action-Conditioned Generation for Policy Learning [53.77164209208635]
  We introduce WorldEcho, which probes action following over a broader action distribution using visual integrity and trajectory alignment.<n>We propose WorldSync, which strengthens action following along three axes: distributional coverage, representational grounding, and intervention-effect alignment.<n>Experiments on RoboTwin benchmarks and real-robot tasks show that WorldSync improves WorldEcho metrics and serves as a more reliable simulator for iterative policy improvement.
  arXiv  Detail & Related papers  (2026-08-25T17:59:49Z)
• Counterfactual learning of new adaptive instructional policies using logged data [1.0534609884264237]
  This paper introduces an offline contextual bandit framework that learns new adaptive policies directly from logged interaction data.<n>We show that effective instructional policies can be learned and visualized within seconds of computation.
  arXiv  Detail & Related papers  (2026-06-22T08:28:34Z)
• Data-dependent Exploration for Online Reinforcement Learning from Human Feedback [50.34161049551627]
  Online reinforcement learning from human feedback (RLHF) has emerged as a promising paradigm for aligning large language models (LLMs) by continuously collecting new preference feedback during training.<n>Existing exploration strategies often derive bonuses via on-policy expectations, which are difficult to estimate reliably from the limited historical preference data available during training.<n>We propose data-dependent exploration for preference optimization (DEPO), a simple and scalable method that leverages historical data to construct an extra uncertainty bonus for high-uncertainty regions.
  arXiv  Detail & Related papers  (2026-05-06T03:56:45Z)
• An Efficient Metric for Data Quality Measurement in Imitation Learning [1.5469452301122175]
  Fine-tuning pre-trained policies with end-user demonstrations collected in deployment environments is a promising strategy to address this challenge.<n>Existing automated approaches for curating demonstration data require policy rollouts in the environment.<n>We propose a fast, efficient, and fully automated demonstration ranking metric based on the power spectral density (PSD) of demonstration trajectories.
  arXiv  Detail & Related papers  (2026-05-02T17:16:50Z)
• FAR-Dex: Few-shot Data Augmentation and Adaptive Residual Policy Refinement for Dexterous Manipulation [9.630733148732475]
  We propose FAR-Dex, a hierarchical framework that integrates few-shot data augmentation with adaptive residual refinement to enable robust and precise arm-hand coordination in dexterous tasks.<n>Experiments in both simulation and real-world demonstrate that FAR-Dex improves data quality by 13.4% and task success rates by 7% over state-of-the-art methods.<n>It further achieves over 80% success in real-world tasks, enabling fine-grained dexterous manipulation with strong positional generalization.
  arXiv  Detail & Related papers  (2026-03-11T06:10:03Z)
• Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning [21.705096559151286]
  We show that offline reinforcement learning can be used to harness non-expert data to enhance the performance of imitation learning policies.<n>Our approach shows that broadening the support of the policy distribution can allow imitation algorithms augmented by offline RL to solve tasks robustly.
  arXiv  Detail & Related papers  (2025-10-22T11:43:39Z)
• CUPID: Curating Data your Robot Loves with Influence Functions [29.79350259314518]
  CUPID is a robot data curation method based on a novel influence function-theoretic formulation for imitation learning policies.<n>We use CUPID to curate data by 1) filtering out training demonstrations that harm policy performance and 2) subselecting newly collected trajectories that will most improve the policy.
  arXiv  Detail & Related papers  (2025-06-23T20:49:34Z)
• Reinforcement Learning via Implicit Imitation Guidance [49.88208134736617]
  A natural approach is to incorporate an imitation learning objective, either as regularization during training or to acquire a reference policy.<n>We propose to use prior data solely for guiding exploration via noise added to the policy, sidestepping the need for explicit behavior cloning constraints.<n>Our approach achieves up to 2-3x improvement over prior reinforcement learning from offline methods across seven simulated continuous control tasks.
  arXiv  Detail & Related papers  (2025-06-09T07:32:52Z)
• Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models [71.34520793462069]
  Unsupervised reinforcement learning (RL) aims at pre-training agents that can solve a wide range of downstream tasks in complex environments.<n>We introduce a novel algorithm regularizing unsupervised RL towards imitating trajectories from unlabeled behavior datasets.<n>We demonstrate the effectiveness of this new approach in a challenging humanoid control problem.
  arXiv  Detail & Related papers  (2025-04-15T10:41:11Z)
• Pessimistic Causal Reinforcement Learning with Mediators for Confounded Offline Data [17.991833729722288]
  We propose a novel policy learning algorithm, PESsimistic CAusal Learning (PESCAL) Our key observation is that, by incorporating auxiliary variables that mediate the effect of actions on system dynamics, it is sufficient to learn a lower bound of the mediator distribution function, instead of the Q-function. We provide theoretical guarantees for the algorithms we propose, and demonstrate their efficacy through simulations, as well as real-world experiments utilizing offline datasets from a leading ride-hailing platform.
  arXiv  Detail & Related papers  (2024-03-18T14:51:19Z)
• Latent-Variable Advantage-Weighted Policy Optimization for Offline RL [70.01851346635637]
  offline reinforcement learning methods hold the promise of learning policies from pre-collected datasets without the need to query the environment for new transitions. In practice, offline datasets are often heterogeneous, i.e., collected in a variety of scenarios. We propose to leverage latent-variable policies that can represent a broader class of policy distributions. Our method improves the average performance of the next best-performing offline reinforcement learning methods by 49% on heterogeneous datasets.
  arXiv  Detail & Related papers  (2022-03-16T21:17:03Z)
• A Regularized Implicit Policy for Offline Reinforcement Learning [54.7427227775581]
  offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. We propose a framework that supports learning a flexible yet well-regularized fully-implicit policy. Experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs.
  arXiv  Detail & Related papers  (2022-02-19T20:22:04Z)
• DEALIO: Data-Efficient Adversarial Learning for Imitation from Observation [57.358212277226315]
  In imitation learning from observation IfO, a learning agent seeks to imitate a demonstrating agent using only observations of the demonstrated behavior without access to the control signals generated by the demonstrator. Recent methods based on adversarial imitation learning have led to state-of-the-art performance on IfO problems, but they typically suffer from high sample complexity due to a reliance on data-inefficient, model-free reinforcement learning algorithms. This issue makes them impractical to deploy in real-world settings, where gathering samples can incur high costs in terms of time, energy, and risk. We propose a more data-efficient IfO algorithm
  arXiv  Detail & Related papers  (2021-03-31T23:46:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.