Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions

• URL: http://arxiv.org/abs/2609.16683v1
• Date: Tue, 15 Sep 2026 06:04:36 GMT
• Title: Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions
• Abstract summary: Learning humanoid-object interaction requires coordinating whole-body balance, locomotion, and dexterous hand contact to control both robot and object motion.<n>We present Weave, a unified framework for learning whole-body dexterous humanoid-object interaction from captured human demonstrations.
• Score: 6.968853416384178
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning humanoid-object interaction requires coordinating whole-body balance, locomotion, and dexterous hand contact to control both robot and object motion. Human demonstrations provide examples of coordinated interaction, but transferring these behaviors to humanoid robots requires learning how to establish and maintain effective contacts under different embodiments and dynamics. We present Weave, a unified framework for learning whole-body dexterous humanoid-object interaction from captured human demonstrations. Weave first converts captured human-object interactions into executable robot-object references through contact-aware retargeting and approach-motion completion. At its core is a contact- and geometry-aware policy that jointly commands 29 body joints and 12 actuated finger joints across multiple objects and interaction sequences. Evaluation across nine objects yields a 92.5% success rate on trained interactions and, without any additional training, 65.0% on sequences never seen during training. We additionally release ~9,000 physically executed rollouts spanning ~23 hours, providing robot-object trajectories with contact annotations for downstream interaction-policy learning and physically consistent HOI motion generation. Project website: https://xiaohu-art.github.io/Weave/

Related papers

• Human2Any: Human-to-Robot Transfer via Constraint-Aware Compositional Planning [17.90707618632327]
  We present Human2Any, a framework for learning reusable object-centric interaction priors from human videos.<n>We validate Human2Any across diverse manipulation settings, including real-world experiments on a Franka tabletop setup and an RBY-1 humanoid mobile robot.
  arXiv  Detail & Related papers  (2026-06-27T08:45:27Z)
• HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos [74.43500240121476]
  We present HumanX, a full-stack framework that compiles human video into generalizable, real-world interaction skills for humanoids.<n>HumanX achieves over 8 times higher generalization success than prior methods.
  arXiv  Detail & Related papers  (2026-02-02T18:53:01Z)
• OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction [76.44108003274955]
  A dominant paradigm for teaching humanoid robots complex skills is to retarget human motions as kinematic references to train reinforcement learning policies.<n>We introduce OmniRetarget, an interaction-preserving data generation engine based on an interaction mesh.<n>By minimizing the Laplacian deformation between the human and robot meshes, OmniRetarget generates kinematically feasible trajectories.
  arXiv  Detail & Related papers  (2025-09-30T17:59:02Z)
• Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesis [51.95817740348585]
  Human-X is a novel framework designed to enable immersive and physically plausible human interactions across diverse entities.<n>Our method jointly predicts actions and reactions in real-time using an auto-regressive reaction diffusion planner.<n>Our framework is validated in real-world applications, including virtual reality interface for human-robot interaction.
  arXiv  Detail & Related papers  (2025-08-04T06:35:48Z)
• RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrations [38.1742893736782]
  RHINO is a general humanoid-human-object interaction framework.<n>It provides a unified view of reactive motion, instruction-based manipulation, and safety concerns.<n>It decouples the interaction process into two levels: 1) a high-level planner inferring human intentions from real-time human behaviors; and 2) a low-level controller achieving reactive motion behaviors and object manipulation skills based on predicted intentions.
  arXiv  Detail & Related papers  (2025-02-18T18:56:41Z)
• CooHOI: Learning Cooperative Human-Object Interaction with Manipulated Object Dynamics [44.30880626337739]
  CooHOI is a framework designed to tackle the challenge of multi-humanoid object transportation problem. A single humanoid character learns to interact with objects through imitation learning from human motion priors. Then, the humanoid learns to collaborate with others by considering the shared dynamics of the manipulated object.
  arXiv  Detail & Related papers  (2024-06-20T17:59:22Z)
• Learning Multimodal Latent Dynamics for Human-Robot Interaction [18.68936554172693]
  This article presents a method for learning well-coordinated Human-Robot Interaction (HRI) from Human-Human Interactions (HHI)<n>We devise a hybrid approach using Hidden Markov Models (HMMs) as the latent space priors for a Variational Autoencoder to model a joint distribution over the interacting agents.<n>We find that users perceive our method as more human-like, timely, and accurate and rank our method with a higher degree of preference over other baselines.
  arXiv  Detail & Related papers  (2023-11-27T23:56:59Z)
• InterControl: Zero-shot Human Interaction Generation by Controlling Every Joint [67.6297384588837]
  We introduce a novel controllable motion generation method, InterControl, to encourage the synthesized motions maintaining the desired distance between joint pairs. We demonstrate that the distance between joint pairs for human-wise interactions can be generated using an off-the-shelf Large Language Model.
  arXiv  Detail & Related papers  (2023-11-27T14:32:33Z)
• HandMeThat: Human-Robot Communication in Physical and Social Environments [73.91355172754717]
  HandMeThat is a benchmark for a holistic evaluation of instruction understanding and following in physical and social environments. HandMeThat contains 10,000 episodes of human-robot interactions. We show that both offline and online reinforcement learning algorithms perform poorly on HandMeThat.
  arXiv  Detail & Related papers  (2023-10-05T16:14:46Z)
• Synthesis and Execution of Communicative Robotic Movements with Generative Adversarial Networks [59.098560311521034]
  We focus on how to transfer on two different robotic platforms the same kinematics modulation that humans adopt when manipulating delicate objects. We choose to modulate the velocity profile adopted by the robots' end-effector, inspired by what humans do when transporting objects with different characteristics. We exploit a novel Generative Adversarial Network architecture, trained with human kinematics examples, to generalize over them and generate new and meaningful velocity profiles.
  arXiv  Detail & Related papers  (2022-03-29T15:03:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.