DexSynRefine: Synthesizing and Refining Human-Object Interaction Motion for Physically Feasible Dexterous Robot Actions

• URL: http://arxiv.org/abs/2605.05925v1
• Date: Thu, 07 May 2026 09:31:43 GMT
• Title: DexSynRefine: Synthesizing and Refining Human-Object Interaction Motion for Physically Feasible Dexterous Robot Actions
• Abstract summary: We present DexSyn, a framework for learning dexterous manipulation from human-object interaction (HOI) data.<n>We also present a contact-and-orientation adaptation module that enables sim-to-real transfer from proprioceptive history.
• Score: 0.5599792629509227
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning dexterous manipulation from human-object interaction (HOI) data is a scalable alternative to teleoperation, but HOI demonstrations are sparse and provide only kinematic motion that is not directly executable under embodiment mismatch and contact-rich dynamics. We present DexSynRefine, a framework with three coupled components: HOI-MMFP, a task- and object-initial-state-conditioned extension of motion manifold primitives that synthesizes coordinated hand-object trajectories from sparse HOI demonstrations; a task-space residual RL policy that physically grounds the synthesized reference while inheriting its kinematic structure; and a contact-and-dynamics adaptation module that enables sim-to-real transfer from proprioceptive history. Across five dexterous manipulation tasks spanning pick-and-place, tool use, and object reorientation, our task-space residual policy outperforms prior action-representation baselines in simulations and transfers to a real robot on all five tasks, improving over kinematic retargeting by 50-70 percentage points.

Related papers

• GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training [20.414478780328437]
  Existing methods often require time-consuming task-specific policy training or rely on rigid trajectory replay.<n>We present textitGenHOI, a framework that enables humanoid robots to perform diverse object-interaction tasks in a zero-shot manner.<n>We validate the proposed framework in extensive simulation and real-world experiments across diverse object-interaction tasks.
  arXiv  Detail & Related papers  (2026-06-11T07:31:05Z)
• X-OP: Cross-Morphology Whole-Body Teleoperation via MPC Retargeting [3.134050607080983]
  Whole-body teleoperation is essential for scalable robot data collection in loco-manipulation tasks.<n>We propose a hierarchical whole-body teleoperation framework driven by a single XR device.<n>We show higher success rates on whole-body control tasks for both a humanoid and a mobile manipulator.
  arXiv  Detail & Related papers  (2026-06-06T01:50:59Z)
• SynAgent: Generalizable Cooperative Humanoid Manipulation via Solo-to-Cooperative Agent Synergy [59.86199649493784]
  Controllable cooperative humanoid manipulation is a fundamental yet challenging problem for embodied intelligence.<n>We present SynAgent, a framework that enables scalable and physically plausible cooperative manipulation.<n>In experiments, SynAgent significantly outperforms existing baselines in both cooperative imitation and trajectory-conditioned control.
  arXiv  Detail & Related papers  (2026-04-20T17:46:20Z)
• SocialMirror: Reconstructing 3D Human Interaction Behaviors from Monocular Videos with Semantic and Geometric Guidance [49.69016078147708]
  Accurately reconstructing human behavior in close-interaction scenarios is crucial for enabling realistic virtual interactions in augmented reality.<n>We propose SocialMirror, a diffusion-based framework that integrates semantic and geometric cues to effectively address these issues.<n>SocialMirror achieves state-of-the-art performance in reconstructing interactive human meshes, demonstrating strong generalization across unseen datasets and in-the-wild scenarios.
  arXiv  Detail & Related papers  (2026-04-15T07:41:52Z)
• MeshMimic: Geometry-Aware Humanoid Motion Learning through 3D Scene Reconstruction [54.36564144414704]
  MeshMimic is an innovative framework that bridges 3D scene reconstruction and embodied intelligence to enable humanoid robots to learn coupled "motion-terrain" interactions directly from video.<n>By leveraging state-of-the-art 3D vision models, our framework precisely segments and reconstructs both human trajectories and the underlying 3D geometry of terrains and objects.
  arXiv  Detail & Related papers  (2026-02-17T17:09:45Z)
• ExoGS: A 4D Real-to-Sim-to-Real Framework for Scalable Manipulation Data Collection [27.18540416686717]
  ExoGS is a robot-free 4D Real-to-Sim-to-Real framework.<n>It captures both static environments and dynamic interactions in the real world and transfers them seamlessly to a simulated environment.<n>It provides a new solution for scalable manipulation data collection and policy learning.
  arXiv  Detail & Related papers  (2026-01-26T16:04:12Z)
• Decoupled Generative Modeling for Human-Object Interaction Synthesis [35.78156236836254]
  Existing approaches often require manually specified intermediate waypoints and place all optimization objectives on a single network.<n>We propose Decoupled Generative Modeling for Human-Object Interaction Synthesis (DecHOI)<n>A trajectory generator first produces human and object trajectories without prescribed waypoints, and an action generator conditions on these paths to synthesize detailed motions.
  arXiv  Detail & Related papers  (2025-12-22T05:33:59Z)
• One-Shot Real-World Demonstration Synthesis for Scalable Bimanual Manipulation [45.00986521352502]
  BiDemoSyn is a framework that synthesizes contact-rich, physically feasible bimanual demonstrations from a single real-world example.<n>We show that policies trained on BiDemoSyn data generalize robustly to novel object poses and shapes.<n> policies trained on BiDemoSyn data exhibit zero-shot cross-embodiment transfer to new robotic platforms.
  arXiv  Detail & Related papers  (2025-12-10T03:48:16Z)
• OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction [76.44108003274955]
  A dominant paradigm for teaching humanoid robots complex skills is to retarget human motions as kinematic references to train reinforcement learning policies.<n>We introduce OmniRetarget, an interaction-preserving data generation engine based on an interaction mesh.<n>By minimizing the Laplacian deformation between the human and robot meshes, OmniRetarget generates kinematically feasible trajectories.
  arXiv  Detail & Related papers  (2025-09-30T17:59:02Z)
• Human-Object Interaction via Automatically Designed VLM-Guided Motion Policy [30.43930233035367]
  We introduce the first unified physics-based HO framework that leverages Vision-Language Models (VLMs)<n>We introduce VLM-Guided Relative Movement Dynamics (RMD), a fine-temporal bipartite motion representation that automatically constructs goal states and reward functions for reinforcement learning.<n>To support our methodology, we present Interplay, a novel dataset with thousands of long-horizon static and dynamic interaction plans.
  arXiv  Detail & Related papers  (2025-03-24T05:18:04Z)
• Controllable Human-Object Interaction Synthesis [77.56877961681462]
  We propose Controllable Human-Object Interaction Synthesis (CHOIS) to generate synchronized object motion and human motion in 3D scenes. Here, language descriptions inform style and intent, and waypoints, which can be effectively extracted from high-level planning, ground the motion in the scene. Our module seamlessly integrates with a path planning module, enabling the generation of long-term interactions in 3D environments.
  arXiv  Detail & Related papers  (2023-12-06T21:14:20Z)
• NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesis [21.650091018774972]
  We create a neural interaction field attached to a specific object, which outputs the distance to the valid interaction manifold given a human pose as input. This interaction field guides the sampling of an object-conditioned human motion diffusion model. We synthesize realistic motions for sitting and lifting with several objects, outperforming alternative approaches in terms of motion quality and successful action completion.
  arXiv  Detail & Related papers  (2023-07-14T17:59:38Z)
• Nonprehensile Riemannian Motion Predictive Control [57.295751294224765]
  We introduce a novel Real-to-Sim reward analysis technique to reliably imagine and predict the outcome of taking possible actions for a real robotic platform. We produce a closed-loop controller to reactively push objects in a continuous action space. We observe that RMPC is robust in cluttered as well as occluded environments and outperforms the baselines.
  arXiv  Detail & Related papers  (2021-11-15T18:50:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.