ThermoHands: A Benchmark for 3D Hand Pose Estimation from Egocentric Thermal Images

• URL: http://arxiv.org/abs/2403.09871v4
• Date: Fri, 15 Nov 2024 16:01:39 GMT
• Title: ThermoHands: A Benchmark for 3D Hand Pose Estimation from Egocentric Thermal Images
• Authors: Fangqiang Ding, Yunzhou Zhu, Xiangyu Wen, Gaowen Liu, Chris Xiaoxuan Lu,
• Abstract summary: We present ThermoHands, the first benchmark focused on thermal image-based egocentric 3D hand pose estimation. We introduce a new baseline method, TherFormer, utilizing dual transformer modules for effective egocentric 3D hand pose estimation in thermal imagery.
• Score: 12.887546538760436
• License:
• Abstract: Designing egocentric 3D hand pose estimation systems that can perform reliably in complex, real-world scenarios is crucial for downstream applications. Previous approaches using RGB or NIR imagery struggle in challenging conditions: RGB methods are susceptible to lighting variations and obstructions like handwear, while NIR techniques can be disrupted by sunlight or interference from other NIR-equipped devices. To address these limitations, we present ThermoHands, the first benchmark focused on thermal image-based egocentric 3D hand pose estimation, demonstrating the potential of thermal imaging to achieve robust performance under these conditions. The benchmark includes a multi-view and multi-spectral dataset collected from 28 subjects performing hand-object and hand-virtual interactions under diverse scenarios, accurately annotated with 3D hand poses through an automated process. We introduce a new baseline method, TherFormer, utilizing dual transformer modules for effective egocentric 3D hand pose estimation in thermal imagery. Our experimental results highlight TherFormer's leading performance and affirm thermal imaging's effectiveness in enabling robust 3D hand pose estimation in adverse conditions.

Related papers

• WiLoR: End-to-end 3D Hand Localization and Reconstruction in-the-wild [53.288327629960364]
  We present a data-driven pipeline for efficient multi-hand reconstruction in the wild. The proposed pipeline is composed of two components: a real-time fully convolutional hand localization and a high-fidelity transformer-based 3D hand reconstruction model. Our approach outperforms previous methods in both efficiency and accuracy on popular 2D and 3D benchmarks.
  arXiv  Detail & Related papers  (2024-09-18T18:46:51Z)
• SHARP: Segmentation of Hands and Arms by Range using Pseudo-Depth for Enhanced Egocentric 3D Hand Pose Estimation and Action Recognition [5.359837526794863]
  Hand pose represents key information for action recognition in the egocentric perspective. We propose to improve egocentric 3D hand pose estimation based on RGB frames only by using pseudo-depth images.
  arXiv  Detail & Related papers  (2024-08-19T14:30:29Z)
• 3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera [59.846927201816776]
  This paper introduces the first framework for 3D tracking of two fast-moving and interacting hands from a single monocular event camera. Our approach tackles the left-right hand ambiguity with a novel semi-supervised feature-wise attention mechanism and integrates an intersection loss to fix hand collisions.
  arXiv  Detail & Related papers  (2023-12-21T18:59:57Z)
• 3D Interacting Hand Pose Estimation by Hand De-occlusion and Removal [85.30756038989057]
  Estimating 3D interacting hand pose from a single RGB image is essential for understanding human actions. We propose to decompose the challenging interacting hand pose estimation task and estimate the pose of each hand separately. Experiments show that the proposed method significantly outperforms previous state-of-the-art interacting hand pose estimation approaches.
  arXiv  Detail & Related papers  (2022-07-22T13:04:06Z)
• Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements [96.40125818594952]
  We make the first attempt to reconstruct 3D interacting hands from monocular single RGB images. Our method can generate 3D hand meshes with both precise 3D poses and minimal collisions.
  arXiv  Detail & Related papers  (2021-11-01T08:24:10Z)
• 3D Hand Pose and Shape Estimation from RGB Images for Improved Keypoint-Based Hand-Gesture Recognition [25.379923604213626]
  This paper presents a keypoint-based end-to-end framework for the 3D hand and pose estimation. It is successfully applied to the hand-gesture recognition task as a study case.
  arXiv  Detail & Related papers  (2021-09-28T17:07:43Z)
• RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video [76.86512780916827]
  We present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline.
  arXiv  Detail & Related papers  (2021-06-22T12:53:56Z)
• Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interaction [137.28465645405655]
  HANDS'19 is a challenge to evaluate the abilities of current 3D hand pose estimators (HPEs) to interpolate and extrapolate the poses of a training set. We show that the accuracy of state-of-the-art methods can drop, and that they fail mostly on poses absent from the training set.
  arXiv  Detail & Related papers  (2020-03-30T19:28:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.