Learning Sim-Grounded Policies for Bimanual Rope Manipulation from Human Teleoperation Data

• URL: http://arxiv.org/abs/2605.16043v1
• Date: Fri, 15 May 2026 15:15:12 GMT
• Title: Learning Sim-Grounded Policies for Bimanual Rope Manipulation from Human Teleoperation Data
• Abstract summary: We investigate whether the lack of generalization in egocentric visual policies for the knot-untangling task stems from the observation space itself.<n>We compare two Action Chunking with Transformers policies trained on the same bimanual teleoperation data.<n>A state-based policy outperforms its visual counterpart with a 30.8% reduction in L1 error when predicting the initial grasp-and-pull action.
• Score: 12.632998789712547
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deformable Linear Objects (DLOs) such as ropes and cables are widely encountered in both household and industrial applications, yet remain challenging to manipulate due to their infinite-dimensional configuration space and frequent self-occlusion. Imitation learning from teleoperation offers a practical path to bimanual DLO manipulation, but its scalability is limited by human effort, making the choice of observation space critical for generalization from small datasets. In this study, we investigate whether the lack of generalization in egocentric visual policies for the knot-untangling task stems from the observation space itself, rather than from the policy architecture or data scale. We compare two Action Chunking with Transformers policies trained on the same bimanual teleoperation data: a vision-based policy conditioned on two egocentric RGB streams from wrist-mounted cameras, and a state-based policy conditioned on the DLO's 3D particle state, extracted from an initial observation via multi-view fusion and evolved in a particle-based eXtended Position-Based Dynamics simulation. Evaluated open-loop on an unseen rope configuration, the state-based policy outperforms its visual counterpart with a 30.8% reduction in L1 error when predicting the initial grasp-and-pull action, quantifying the observability gap between pixels and physics-consistent state, and pointing toward more data-efficient robot learning for the DLO manipulation task from limited human demonstrations.

Related papers

• AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding [55.535374902204296]
  AffordanceVLA is a unified framework that introduces structured affordance forecasting as a task-oriented intermediate representation.<n>We show that AffordanceVLA achieves strong performance across diverse manipulation scenarios.
  arXiv  Detail & Related papers  (2026-06-04T13:28:51Z)
• VISTA: Vision-Grounded and Physics-Validated Adaptation of UMI data for VLA Training [52.05483137072975]
  Universal Manipulation Interface (UMI) enables scalable real-world robot data collection without hardware-specific teleoperation.<n>We present VISTA, a framework that bridges this dual gap through three synergistic components.<n>We release the physical-validation pipeline, UMI-VQA, validated trajectory data, and the pre-trained model for the community.
  arXiv  Detail & Related papers  (2026-06-03T10:38:45Z)
• WARPED: Wrist-Aligned Rendering for Robot Policy Learning from Egocentric Human Demonstrations [10.024841990710177]
  WARPED is a framework designed to synthesize realistic wrist-view observations from human demonstration videos.<n>A hand-object interaction pipeline is then employed to track the hand and manipulated object and retarget the trajectories to a robotic end-effector.<n>We demonstrate that WARPED achieves success rates comparable to policies trained on teleoperated demonstration data for five tabletop manipulation tasks.
  arXiv  Detail & Related papers  (2026-04-12T20:40:59Z)
• Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation [57.28703268044067]
  We propose Cortical Policy, a novel dual-stream view transformer for robotic manipulation.<n>Our framework offers a fresh perspective for robotic manipulation and holds potential for broader application in vision-based robot control.
  arXiv  Detail & Related papers  (2026-03-22T04:18:54Z)
• ULTRA: Unified Multimodal Control for Autonomous Humanoid Whole-Body Loco-Manipulation [55.467742403416175]
  We introduce a physics-driven neural algorithm that translates large-scale motion capture to humanoid embodiments.<n>We learn a unified multimodal controller that supports both dense references and sparse task specifications.<n>Results show that ULTRA generalizes to autonomous, goal-conditioned whole-body loco-manipulation from egocentric perception.
  arXiv  Detail & Related papers  (2026-03-03T18:59:29Z)
• UniStateDLO: Unified Generative State Estimation and Tracking of Deformable Linear Objects Under Occlusion for Constrained Manipulation [45.803487170590266]
  Perception of deformable linear objects (DLOs) is the cornerstone for successful downstream manipulation.<n>This paper presents UniStateDLO, the first complete DLO perception pipeline with deep-learning methods.
  arXiv  Detail & Related papers  (2025-12-19T16:35:02Z)
• Opening the Sim-to-Real Door for Humanoid Pixel-to-Action Policy Transfer [59.02729900344616]
  GPU-accelerated, photorealistic simulation has opened a scalable data-generation path for robot learning.<n>We develop a teacher-student-bootstrap learning framework for vision-based humanoid loco-manipulation.<n>This represents the first humanoid sim-to-real policy capable of diverse articulated loco-manipulation using pure RGB perception.
  arXiv  Detail & Related papers  (2025-11-30T20:07:13Z)
• R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulation [74.41728218960465]
  We propose a real-to-real 3D data generation framework (R2RGen) that directly augments the pointcloud observation-action pairs to generate real-world data.<n>R2RGen substantially enhances data efficiency on extensive experiments and demonstrates strong potential for scaling and application on mobile manipulation.
  arXiv  Detail & Related papers  (2025-10-09T17:55:44Z)
• A Distributional Treatment of Real2Sim2Real for Object-Centric Agent Adaptation in Vision-Driven Deformable Linear Object Manipulation [16.26835655544884]
  We present an integrated framework for manipulating deformable linear objects (DLOs) based on visual perception.<n>We compute the posterior distributions for the physical parameters using which we can approximately simulate the behaviour of each specific DLO.<n>We then study the implications of the resulting domain distributions in sim-based policy learning and real-world performance.
  arXiv  Detail & Related papers  (2025-02-25T20:01:06Z)
• Bridging the Gap to Real-World Object-Centric Learning [66.55867830853803]
  We show that reconstructing features from models trained in a self-supervised manner is a sufficient training signal for object-centric representations to arise in a fully unsupervised way. Our approach, DINOSAUR, significantly out-performs existing object-centric learning models on simulated data.
  arXiv  Detail & Related papers  (2022-09-29T15:24:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.