Physical Interaction: Reconstructing Hand-object Interactions with Physics

• URL: http://arxiv.org/abs/2209.10833v1
• Date: Thu, 22 Sep 2022 07:41:31 GMT
• Title: Physical Interaction: Reconstructing Hand-object Interactions with Physics
• Authors: Haoyu Hu, Xinyu Yi, Hao Zhang, Jun-Hai Yong, Feng Xu
• Abstract summary: The paper proposes a physics-based method to better solve the ambiguities in the reconstruction. It first proposes a force-based dynamic model of the in-hand object, which recovers the unobserved contacts and also solves for plausible contact forces. Experiments show that the proposed technique reconstructs both physically plausible and more accurate hand-object interaction.
• Score: 17.90852804328213
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Single view-based reconstruction of hand-object interaction is challenging due to the severe observation missing caused by occlusions. This paper proposes a physics-based method to better solve the ambiguities in the reconstruction. It first proposes a force-based dynamic model of the in-hand object, which not only recovers the unobserved contacts but also solves for plausible contact forces. Next, a confidence-based slide prevention scheme is proposed, which combines both the kinematic confidences and the contact forces to jointly model static and sliding contact motion. Qualitative and quantitative experiments show that the proposed technique reconstructs both physically plausible and more accurate hand-object interaction and estimates plausible contact forces in real-time with a single RGBD sensor.

Related papers

• Dynamic Reconstruction of Hand-Object Interaction with Distributed Force-aware Contact Representation [52.36691633451968]
  ViTaM-D is a visual-tactile framework for dynamic hand-object interaction reconstruction. DF-Field is a distributed force-aware contact representation model. Our results highlight the superior performance of ViTaM-D in both rigid and deformable object reconstruction.
  arXiv  Detail & Related papers  (2024-11-14T16:29:45Z)
• Physics-aware Hand-object Interaction Denoising [21.3448097349759]
  We introduce two learned loss terms that explicitly capture two crucial aspects of physical plausibility. These terms are used to train a physically-aware de-noising network. Our approach significantly improves both fine-grained physical plausibility and overall pose accuracy, surpassing current state-of-the-art de-noising methods.
  arXiv  Detail & Related papers  (2024-05-19T08:24:34Z)
• Hand-Object Interaction Controller (HOIC): Deep Reinforcement Learning for Reconstructing Interactions with Physics [12.443255379595278]
  We reconstruct hand manipulating motion with a single RGBD camera by a novel deep reinforcement learning method. We propose object compensation control which establishes direct object control to make the network training more stable. By leveraging the compensation force and torque, we seamlessly upgrade the simple point contact model to a more physical-plausible surface contact model.
  arXiv  Detail & Related papers  (2024-05-04T14:32:13Z)
• PhyRecon: Physically Plausible Neural Scene Reconstruction [81.73129450090684]
  We introduce PHYRECON, the first approach to leverage both differentiable rendering and differentiable physics simulation to learn implicit surface representations. Central to this design is an efficient transformation between SDF-based implicit representations and explicit surface points. Our results also exhibit superior physical stability in physical simulators, with at least a 40% improvement across all datasets.
  arXiv  Detail & Related papers  (2024-04-25T15:06:58Z)
• DeepSimHO: Stable Pose Estimation for Hand-Object Interaction via Physics Simulation [81.11585774044848]
  We present DeepSimHO, a novel deep-learning pipeline that combines forward physics simulation and backward gradient approximation with a neural network. Our method noticeably improves the stability of the estimation and achieves superior efficiency over test-time optimization.
  arXiv  Detail & Related papers  (2023-10-11T05:34:36Z)
• Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB Video [104.69686024776396]
  Reconstructing interacting hands from monocular RGB data is a challenging task, as it involves many interfering factors. Previous works only leverage information from a single RGB image without modeling their physically plausible relation. In this work, we are dedicated to explicitly exploiting spatial-temporal information to achieve better interacting hand reconstruction.
  arXiv  Detail & Related papers  (2023-08-08T06:16:37Z)
• Stability-driven Contact Reconstruction From Monocular Color Images [7.427212296770506]
  Physical contact provides additional constraints for hand-object state reconstruction. Existing methods optimize the hand-object contact driven by distance threshold or prior from contact-labeled datasets. Our key idea is to reconstruct the contact pattern directly from monocular images, and then utilize the physical stability criterion in the simulation to optimize it.
  arXiv  Detail & Related papers  (2022-05-02T12:23:06Z)
• Contact-Aware Retargeting of Skinned Motion [49.71236739408685]
  This paper introduces a motion estimation method that preserves self-contacts and prevents interpenetration. The method identifies self-contacts and ground contacts in the input motion, and optimize the motion to apply to the output skeleton. In experiments, our results quantitatively outperform previous methods and we conduct a user study where our retargeted motions are rated as higher-quality than those produced by recent works.
  arXiv  Detail & Related papers  (2021-09-15T17:05:02Z)
• Physically Plausible Pose Refinement using Fully Differentiable Forces [68.8204255655161]
  We propose an end-to-end differentiable model that refines pose estimates by learning the forces experienced by the object. By matching the learned net force to an estimate of net force based on finite differences of position, this model is able to find forces that accurately describe the movement of the object. We show this model successfully corrects poses and finds contact maps that better match the ground truth, despite not using any RGB or depth image data.
  arXiv  Detail & Related papers  (2021-05-17T23:33:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.