Physically Plausible Full-Body Hand-Object Interaction Synthesis

• URL: http://arxiv.org/abs/2309.07907v1
• Date: Thu, 14 Sep 2023 17:55:18 GMT
• Title: Physically Plausible Full-Body Hand-Object Interaction Synthesis
• Authors: Jona Braun, Sammy Christen, Muhammed Kocabas, Emre Aksan, Otmar Hilliges
• Abstract summary: We propose a physics-based method for synthesizing dexterous hand-object interactions in a full-body setting. Existing methods often focus on isolated segments of the interaction process and rely on data-driven techniques that may result in artifacts.
• Score: 32.83908152822006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a physics-based method for synthesizing dexterous hand-object interactions in a full-body setting. While recent advancements have addressed specific facets of human-object interactions, a comprehensive physics-based approach remains a challenge. Existing methods often focus on isolated segments of the interaction process and rely on data-driven techniques that may result in artifacts. In contrast, our proposed method embraces reinforcement learning (RL) and physics simulation to mitigate the limitations of data-driven approaches. Through a hierarchical framework, we first learn skill priors for both body and hand movements in a decoupled setting. The generic skill priors learn to decode a latent skill embedding into the motion of the underlying part. A high-level policy then controls hand-object interactions in these pretrained latent spaces, guided by task objectives of grasping and 3D target trajectory following. It is trained using a novel reward function that combines an adversarial style term with a task reward, encouraging natural motions while fulfilling the task incentives. Our method successfully accomplishes the complete interaction task, from approaching an object to grasping and subsequent manipulation. We compare our approach against kinematics-based baselines and show that it leads to more physically plausible motions.

Related papers

• BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects [70.20706475051347]
  BimArt is a novel generative approach for synthesizing 3D bimanual hand interactions with articulated objects. We first generate distance-based contact maps conditioned on the object trajectory with an articulation-aware feature representation. The learned contact prior is then used to guide our hand motion generator, producing diverse and realistic bimanual motions for object movement and articulation.
  arXiv  Detail & Related papers  (2024-12-06T14:23:56Z)
• AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents [58.807802111818994]
  We propose AnySkill, a novel hierarchical method that learns physically plausible interactions following open-vocabulary instructions. Our approach begins by developing a set of atomic actions via a low-level controller trained via imitation learning. An important feature of our method is the use of image-based rewards for the high-level policy, which allows the agent to learn interactions with objects without manual reward engineering.
  arXiv  Detail & Related papers  (2024-03-19T15:41:39Z)
• Controllable Human-Object Interaction Synthesis [77.56877961681462]
  We propose Controllable Human-Object Interaction Synthesis (CHOIS) to generate synchronized object motion and human motion in 3D scenes. Here, language descriptions inform style and intent, and waypoints, which can be effectively extracted from high-level planning, ground the motion in the scene. Our module seamlessly integrates with a path planning module, enabling the generation of long-term interactions in 3D environments.
  arXiv  Detail & Related papers  (2023-12-06T21:14:20Z)
• CG-HOI: Contact-Guided 3D Human-Object Interaction Generation [29.3564427724612]
  We propose CG-HOI, the first method to generate dynamic 3D human-object interactions (HOIs) from text. We model the motion of both human and object in an interdependent fashion, as semantically rich human motion rarely happens in isolation. We show that our joint contact-based human-object interaction approach generates realistic and physically plausible sequences.
  arXiv  Detail & Related papers  (2023-11-27T18:59:10Z)
• ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulation [29.999224233718927]
  ArtiGrasp is a method to synthesize bi-manual hand-object interactions that include grasping and articulation. Our framework unifies grasping and articulation within a single policy guided by a single hand pose reference. We show that our method can generate motions with noisy hand-object pose estimates from an off-the-shelf image-based regressor.
  arXiv  Detail & Related papers  (2023-09-07T17:53:20Z)
• InterTracker: Discovering and Tracking General Objects Interacting with Hands in the Wild [40.489171608114574]
  Existing methods rely on frame-based detectors to locate interacting objects. We propose to leverage hand-object interaction to track interactive objects. Our proposed method outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-06T09:09:17Z)
• Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations [61.659439423703155]
  TOHO: Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations. Our method generates continuous motions that are parameterized only by the temporal coordinate. This work takes a step further toward general human-scene interaction simulation.
  arXiv  Detail & Related papers  (2023-03-23T09:31:56Z)
• D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions [47.55376158184854]
  We introduce the dynamic synthesis grasp task. Given an object with a known 6D pose and a grasp reference, our goal is to generate motions that move the object to a target 6D pose. A hierarchical approach decomposes the task into low-level grasping and high-level motion synthesis.
  arXiv  Detail & Related papers  (2021-12-01T17:04:39Z)
• Contact-Aware Retargeting of Skinned Motion [49.71236739408685]
  This paper introduces a motion estimation method that preserves self-contacts and prevents interpenetration. The method identifies self-contacts and ground contacts in the input motion, and optimize the motion to apply to the output skeleton. In experiments, our results quantitatively outperform previous methods and we conduct a user study where our retargeted motions are rated as higher-quality than those produced by recent works.
  arXiv  Detail & Related papers  (2021-09-15T17:05:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.