Nonrigid Object Contact Estimation With Regional Unwrapping Transformer

• URL: http://arxiv.org/abs/2308.14074v2
• Date: Wed, 30 Aug 2023 13:40:21 GMT
• Title: Nonrigid Object Contact Estimation With Regional Unwrapping Transformer
• Authors: Wei Xie, Zimeng Zhao, Shiying Li, Binghui Zuo, Yangang Wang
• Abstract summary: Acquiring contact patterns between hands and nonrigid objects is a common concern in the vision and robotics community. Existing learning-based methods focus more on contact with rigid ones from monocular images. We propose a novel hand-object contact representation called RUPs, which unwraps the roughly estimated hand-object surfaces as multiple high-resolution 2D regional profiles.
• Score: 16.988812837693203
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Acquiring contact patterns between hands and nonrigid objects is a common concern in the vision and robotics community. However, existing learning-based methods focus more on contact with rigid ones from monocular images. When adopting them for nonrigid contact, a major problem is that the existing contact representation is restricted by the geometry of the object. Consequently, contact neighborhoods are stored in an unordered manner and contact features are difficult to align with image cues. At the core of our approach lies a novel hand-object contact representation called RUPs (Region Unwrapping Profiles), which unwrap the roughly estimated hand-object surfaces as multiple high-resolution 2D regional profiles. The region grouping strategy is consistent with the hand kinematic bone division because they are the primitive initiators for a composite contact pattern. Based on this representation, our Regional Unwrapping Transformer (RUFormer) learns the correlation priors across regions from monocular inputs and predicts corresponding contact and deformed transformations. Our experiments demonstrate that the proposed framework can robustly estimate the deformed degrees and deformed transformations, which makes it suitable for both nonrigid and rigid contact.

Related papers

• RegionGrasp: A Novel Task for Contact Region Controllable Hand Grasp Generation [35.11194409871017]
  RegionGrasp-CVAE is proposed to generate plausible hand grasps of 3D objects. Condition encoder O-Enc and pretraining strategy O-Enc are used. HoINet is introduced to encode hand-object interaction features.
  arXiv  Detail & Related papers  (2024-10-10T14:52:30Z)
• DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image [98.29284902879652]
  We present DICE, the first end-to-end method for Deformation-aware hand-face Interaction reCovEry from a single image. It features disentangling the regression of local deformation fields and global mesh locations into two network branches. It achieves state-of-the-art performance on a standard benchmark and in-the-wild data in terms of accuracy and physical plausibility.
  arXiv  Detail & Related papers  (2024-06-26T00:08:29Z)
• Pose Priors from Language Models [74.61186408764559]
  We present a zero-shot pose optimization method that enforces accurate physical contact constraints. Our method produces surprisingly compelling pose reconstructions of people in close contact. Unlike previous approaches, our method provides a unified framework for resolving self-contact and person-to-person contact.
  arXiv  Detail & Related papers  (2024-05-06T17:59:36Z)
• Decaf: Monocular Deformation Capture for Face and Hand Interactions [77.75726740605748]
  This paper introduces the first method that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system.
  arXiv  Detail & Related papers  (2023-09-28T17:59:51Z)
• Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images [59.49985837246644]
  We show how to model contacts in an explicit way to benefit the implicit reconstruction of hand-held objects. In the first part, we propose a new subtask of directly estimating 3D hand-object contacts from a single image. In the second part, we introduce a novel method to diffuse estimated contact states from the hand mesh surface to nearby 3D space.
  arXiv  Detail & Related papers  (2023-05-31T17:59:26Z)
• Integrated Object Deformation and Contact Patch Estimation from Visuo-Tactile Feedback [8.420670642409219]
  We propose a representation that jointly models object deformations and contact patches from visuo-tactile feedback. We propose a neural network architecture to learn a NDCF, and train it using simulated data. We demonstrate that the learned NDCF transfers directly to the real-world without the need for fine-tuning.
  arXiv  Detail & Related papers  (2023-05-23T18:53:24Z)
• Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing [36.609644278386135]
  We present Neural Contact Fields, a method that brings together neural fields and tactile sensing to address the problem of tracking extrinsic contact between object and environment. Knowing where the external contact occurs is a first step towards methods that can actively control it in facilitating downstream manipulation tasks.
  arXiv  Detail & Related papers  (2022-10-17T17:52:43Z)
• Contact2Grasp: 3D Grasp Synthesis via Hand-Object Contact Constraint [18.201389966034263]
  3D grasp synthesis generates grasping poses given an input object. We introduce an intermediate variable for grasp contact areas to constrain the grasp generation. Our method outperforms state-of-the-art methods regarding grasp generation on various metrics.
  arXiv  Detail & Related papers  (2022-10-17T16:39:25Z)
• Stability-driven Contact Reconstruction From Monocular Color Images [7.427212296770506]
  Physical contact provides additional constraints for hand-object state reconstruction. Existing methods optimize the hand-object contact driven by distance threshold or prior from contact-labeled datasets. Our key idea is to reconstruct the contact pattern directly from monocular images, and then utilize the physical stability criterion in the simulation to optimize it.
  arXiv  Detail & Related papers  (2022-05-02T12:23:06Z)
• Contact-Aware Retargeting of Skinned Motion [49.71236739408685]
  This paper introduces a motion estimation method that preserves self-contacts and prevents interpenetration. The method identifies self-contacts and ground contacts in the input motion, and optimize the motion to apply to the output skeleton. In experiments, our results quantitatively outperform previous methods and we conduct a user study where our retargeted motions are rated as higher-quality than those produced by recent works.
  arXiv  Detail & Related papers  (2021-09-15T17:05:02Z)
• 3D Shape Reconstruction from Vision and Touch [62.59044232597045]
  In 3D shape reconstruction, the complementary fusion of visual and haptic modalities remains largely unexplored. We introduce a dataset of simulated touch and vision signals from the interaction between a robotic hand and a large array of 3D objects.
  arXiv  Detail & Related papers  (2020-07-07T20:20:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.