ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction

• URL: http://arxiv.org/abs/2303.05938v1
• Date: Fri, 10 Mar 2023 14:19:02 GMT
• Title: ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction
• Authors: Zhengdi Yu, Shaoli Huang, Chen Fang, Toby P. Breckon, Jue Wang
• Abstract summary: We present ACR (Attention Collaboration-based Regressor), which makes the first attempt to reconstruct hands in arbitrary scenarios. We evaluate our method on various types of hand reconstruction datasets.
• Score: 30.073586754012645
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reconstructing two hands from monocular RGB images is challenging due to frequent occlusion and mutual confusion. Existing methods mainly learn an entangled representation to encode two interacting hands, which are incredibly fragile to impaired interaction, such as truncated hands, separate hands, or external occlusion. This paper presents ACR (Attention Collaboration-based Regressor), which makes the first attempt to reconstruct hands in arbitrary scenarios. To achieve this, ACR explicitly mitigates interdependencies between hands and between parts by leveraging center and part-based attention for feature extraction. However, reducing interdependence helps release the input constraint while weakening the mutual reasoning about reconstructing the interacting hands. Thus, based on center attention, ACR also learns cross-hand prior that handle the interacting hands better. We evaluate our method on various types of hand reconstruction datasets. Our method significantly outperforms the best interacting-hand approaches on the InterHand2.6M dataset while yielding comparable performance with the state-of-the-art single-hand methods on the FreiHand dataset. More qualitative results on in-the-wild and hand-object interaction datasets and web images/videos further demonstrate the effectiveness of our approach for arbitrary hand reconstruction. Our code is available at https://github.com/ZhengdiYu/Arbitrary-Hands-3D-Reconstruction.

Related papers

• OmniHands: Towards Robust 4D Hand Mesh Recovery via A Versatile Transformer [35.983309206845036]
  We introduce OmniHands, a universal approach to recovering interactive hand meshes and their relative movement from monocular or multi-view inputs. We develop a universal architecture with novel tokenization and contextual feature fusion strategies. The efficacy of our approach is validated on several benchmark datasets.
  arXiv  Detail & Related papers  (2024-05-30T17:59:02Z)
• Two Hands Are Better Than One: Resolving Hand to Hand Intersections via Occupancy Networks [33.9893684177763]
  Self-occlusions and finger articulation pose a significant problem to estimation. We exploit an occupancy network that represents the hand's volume as a continuous manifold. We design an intersection loss function to minimize the likelihood of hand-to-point intersections.
  arXiv  Detail & Related papers  (2024-04-08T11:32:26Z)
• Exploiting Spatial-Temporal Context for Interacting Hand Reconstruction on Monocular RGB Video [104.69686024776396]
  Reconstructing interacting hands from monocular RGB data is a challenging task, as it involves many interfering factors. Previous works only leverage information from a single RGB image without modeling their physically plausible relation. In this work, we are dedicated to explicitly exploiting spatial-temporal information to achieve better interacting hand reconstruction.
  arXiv  Detail & Related papers  (2023-08-08T06:16:37Z)
• Deformer: Dynamic Fusion Transformer for Robust Hand Pose Estimation [59.3035531612715]
  Existing methods often struggle to generate plausible hand poses when the hand is heavily occluded or blurred. In videos, the movements of the hand allow us to observe various parts of the hand that may be occluded or blurred in a single frame. We propose the Deformer: a framework that implicitly reasons about the relationship between hand parts within the same image.
  arXiv  Detail & Related papers  (2023-03-09T02:24:30Z)
• Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapes [58.551154822792284]
  Implicit Two Hands (Im2Hands) is the first neural implicit representation of two interacting hands. Im2Hands can produce fine-grained geometry of two hands with high hand-to-hand and hand-to-image coherency. We experimentally demonstrate the effectiveness of Im2Hands on two-hand reconstruction in comparison to related methods.
  arXiv  Detail & Related papers  (2023-02-28T06:38:25Z)
• Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image [30.24438569170251]
  We propose a decoupled iterative refinement framework to achieve pixel-alignment hand reconstruction. Our method outperforms all existing two-hand reconstruction methods by a large margin on the InterHand2.6M dataset.
  arXiv  Detail & Related papers  (2023-02-05T15:46:57Z)
• 3D Interacting Hand Pose Estimation by Hand De-occlusion and Removal [85.30756038989057]
  Estimating 3D interacting hand pose from a single RGB image is essential for understanding human actions. We propose to decompose the challenging interacting hand pose estimation task and estimate the pose of each hand separately. Experiments show that the proposed method significantly outperforms previous state-of-the-art interacting hand pose estimation approaches.
  arXiv  Detail & Related papers  (2022-07-22T13:04:06Z)
• Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements [96.40125818594952]
  We make the first attempt to reconstruct 3D interacting hands from monocular single RGB images. Our method can generate 3D hand meshes with both precise 3D poses and minimal collisions.
  arXiv  Detail & Related papers  (2021-11-01T08:24:10Z)
• Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion [78.98074380040838]
  We propose to consider hand and object jointly in feature space and explore the reciprocity of the two branches. We employ an auxiliary depth estimation module to augment the input RGB image with the estimated depth map. Our approach significantly outperforms existing approaches in terms of the reconstruction accuracy of objects.
  arXiv  Detail & Related papers  (2020-06-28T09:50:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.