EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision

• URL: http://arxiv.org/abs/2409.02224v1
• Date: Tue, 3 Sep 2024 18:53:32 GMT
• Title: EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision
• Authors: Yiming Zhao, Taein Kwon, Paul Streli, Marc Pollefeys, Christian Holz,
• Abstract summary: We introduce EgoPressure, a novel dataset of touch contact and pressure interaction from an egocentric perspective. EgoPressure comprises 5.0 hours of touch contact and pressure interaction from 21 participants captured by a moving egocentric camera and 7 stationary Kinect cameras.
• Score: 69.1005706608681
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Estimating touch contact and pressure in egocentric vision is a central task for downstream applications in Augmented Reality, Virtual Reality, as well as many robotic applications, because it provides precise physical insights into hand-object interaction and object manipulation. However, existing contact pressure datasets lack egocentric views and hand poses, which are essential for accurate estimation during in-situ operation, both for AR/VR interaction and robotic manipulation. In this paper, we introduce EgoPressure,a novel dataset of touch contact and pressure interaction from an egocentric perspective, complemented with hand pose meshes and fine-grained pressure intensities for each contact. The hand poses in our dataset are optimized using our proposed multi-view sequence-based method that processes footage from our capture rig of 8 accurately calibrated RGBD cameras. EgoPressure comprises 5.0 hours of touch contact and pressure interaction from 21 participants captured by a moving egocentric camera and 7 stationary Kinect cameras, which provided RGB images and depth maps at 30 Hz. In addition, we provide baselines for estimating pressure with different modalities, which will enable future developments and benchmarking on the dataset. Overall, we demonstrate that pressure and hand poses are complementary, which supports our intention to better facilitate the physical understanding of hand-object interactions in AR/VR and robotics research.

Related papers

• Posture-Informed Muscular Force Learning for Robust Hand Pressure Estimation [6.912016522494431]
  We present PiMForce, a novel framework that enhances hand pressure estimation. Our approach utilizes detailed spatial information from 3D hand poses in conjunction with dynamic muscle activity from sEMG. Our framework enables precise hand pressure estimation in complex and natural interaction scenarios.
  arXiv  Detail & Related papers  (2024-10-31T04:42:43Z)
• Depth Restoration of Hand-Held Transparent Objects for Human-to-Robot Handover [5.329513275750882]
  This paper presents a Hand-Aware Depth Restoration (HADR) method based on creating an implicit neural representation function from a single RGB-D image. The proposed method utilizes hand posture as an important guidance to leverage semantic and geometric information of hand-object interaction. We further develop a real-world human-to-robot handover system based on HADR, demonstrating its potential in human-robot interaction applications.
  arXiv  Detail & Related papers  (2024-08-27T12:25:12Z)
• Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects [89.95728475983263]
  holistic 3Dunderstanding of such interactions from egocentric views is important for tasks in robotics, AR/VR, action recognition and motion generation. We design the HANDS23 challenge based on the AssemblyHands and ARCTIC datasets with carefully designed training and testing splits. Based on the results of the top submitted methods and more recent baselines on the leaderboards, we perform a thorough analysis on 3D hand(-object) reconstruction tasks.
  arXiv  Detail & Related papers  (2024-03-25T05:12:21Z)
• PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images [23.877554759345607]
  Deep models can estimate hand pressure based on a single RGB image. We present a novel approach that enables diverse data to be captured with only an RGB camera and a cooperative participant. We also demonstrate an application of PressureVision++ to mixed reality where pressure estimation allows everyday surfaces to be used as arbitrary touch-sensitive interfaces.
  arXiv  Detail & Related papers  (2023-01-05T21:48:33Z)
• PressureVision: Estimating Hand Pressure from a Single RGB Image [27.449311565446443]
  We explore the possibility of using a conventional RGB camera to infer hand pressure. We collected videos of 36 participants with diverse skin tone applying pressure to an instrumented planar surface. We trained a deep model (PressureVisionNet) to infer a pressure image from a single RGB image.
  arXiv  Detail & Related papers  (2022-03-19T19:54:56Z)
• RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video [76.86512780916827]
  We present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline.
  arXiv  Detail & Related papers  (2021-06-22T12:53:56Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• Where is my hand? Deep hand segmentation for visual self-recognition in humanoid robots [129.46920552019247]
  We propose the use of a Convolution Neural Network (CNN) to segment the robot hand from an image in an egocentric view. We fine-tuned the Mask-RCNN network for the specific task of segmenting the hand of the humanoid robot Vizzy.
  arXiv  Detail & Related papers  (2021-02-09T10:34:32Z)
• Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learning [52.37106940303246]
  We learn a model that maps noisy input hand poses to target virtual poses. The agent is trained in a residual setting by using a model-free hybrid RL+IL approach. We test our framework in two applications that use hand pose estimates for dexterous manipulations: hand-object interactions in VR and hand-object motion reconstruction in-the-wild.
  arXiv  Detail & Related papers  (2020-08-07T17:34:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.