HandDGP: Camera-Space Hand Mesh Prediction with Differentiable Global Positioning

• URL: http://arxiv.org/abs/2407.15844v1
• Date: Mon, 22 Jul 2024 17:59:01 GMT
• Title: HandDGP: Camera-Space Hand Mesh Prediction with Differentiable Global Positioning
• Authors: Eugene Valassakis, Guillermo Garcia-Hernando,
• Abstract summary: We propose an end-to-end solution that addresses the 2D-3D correspondence problem. This solution enables back-propagation from camera space outputs to the rest of the network through a new differentiable global positioning module. We validate the effectiveness of our framework in evaluations against several baselines and state-of-the-art approaches.
• Score: 1.4515751892711464
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predicting camera-space hand meshes from single RGB images is crucial for enabling realistic hand interactions in 3D virtual and augmented worlds. Previous work typically divided the task into two stages: given a cropped image of the hand, predict meshes in relative coordinates, followed by lifting these predictions into camera space in a separate and independent stage, often resulting in the loss of valuable contextual and scale information. To prevent the loss of these cues, we propose unifying these two stages into an end-to-end solution that addresses the 2D-3D correspondence problem. This solution enables back-propagation from camera space outputs to the rest of the network through a new differentiable global positioning module. We also introduce an image rectification step that harmonizes both the training dataset and the input image as if they were acquired with the same camera, helping to alleviate the inherent scale-depth ambiguity of the problem. We validate the effectiveness of our framework in evaluations against several baselines and state-of-the-art approaches across three public benchmarks.

Related papers

• DICE: End-to-end Deformation Capture of Hand-Face Interactions from a Single Image [98.29284902879652]
  We present DICE, the first end-to-end method for Deformation-aware hand-face Interaction reCovEry from a single image. It features disentangling the regression of local deformation fields and global mesh locations into two network branches. It achieves state-of-the-art performance on a standard benchmark and in-the-wild data in terms of accuracy and physical plausibility.
  arXiv  Detail & Related papers  (2024-06-26T00:08:29Z)
• 3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera [59.846927201816776]
  This paper introduces the first framework for 3D tracking of two fast-moving and interacting hands from a single monocular event camera. Our approach tackles the left-right hand ambiguity with a novel semi-supervised feature-wise attention mechanism and integrates an intersection loss to fix hand collisions.
  arXiv  Detail & Related papers  (2023-12-21T18:59:57Z)
• The Change You Want to See (Now in 3D) [65.61789642291636]
  The goal of this paper is to detect what has changed, if anything, between two "in the wild" images of the same 3D scene. We contribute a change detection model that is trained entirely on synthetic data and is class-agnostic. We release a new evaluation dataset consisting of real-world image pairs with human-annotated differences.
  arXiv  Detail & Related papers  (2023-08-21T01:59:45Z)
• Decoupled Iterative Refinement Framework for Interacting Hands Reconstruction from a Single RGB Image [30.24438569170251]
  We propose a decoupled iterative refinement framework to achieve pixel-alignment hand reconstruction. Our method outperforms all existing two-hand reconstruction methods by a large margin on the InterHand2.6M dataset.
  arXiv  Detail & Related papers  (2023-02-05T15:46:57Z)
• Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements [96.40125818594952]
  We make the first attempt to reconstruct 3D interacting hands from monocular single RGB images. Our method can generate 3D hand meshes with both precise 3D poses and minimal collisions.
  arXiv  Detail & Related papers  (2021-11-01T08:24:10Z)
• Learning to Disambiguate Strongly Interacting Hands via Probabilistic Per-pixel Part Segmentation [84.28064034301445]
  Self-similarity, and the resulting ambiguities in assigning pixel observations to the respective hands, is a major cause of the final 3D pose error. We propose DIGIT, a novel method for estimating the 3D poses of two interacting hands from a single monocular image. We experimentally show that the proposed approach achieves new state-of-the-art performance on the InterHand2.6M dataset.
  arXiv  Detail & Related papers  (2021-07-01T13:28:02Z)
• Deep Bingham Networks: Dealing with Uncertainty and Ambiguity in Pose Estimation [74.76155168705975]
  Deep Bingham Networks (DBN) can handle pose-related uncertainties and ambiguities arising in almost all real life applications concerning 3D data. DBN extends the state of the art direct pose regression networks by (i) a multi-hypotheses prediction head which can yield different distribution modes. We propose new training strategies so as to avoid mode or posterior collapse during training and to improve numerical stability.
  arXiv  Detail & Related papers  (2020-12-20T19:20:26Z)
• Two-hand Global 3D Pose Estimation Using Monocular RGB [0.0]
  We tackle the challenging task of estimating global 3D joint locations for both hands via only monocular RGB input images. We propose a novel multi-stage convolutional neural network based pipeline that accurately segments and locates the hands. We present the first work that achieves accurate global 3D hand tracking on both hands using RGB-only inputs.
  arXiv  Detail & Related papers  (2020-06-01T23:53:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.