Complementing Event Streams and RGB Frames for Hand Mesh Reconstruction

• URL: http://arxiv.org/abs/2403.07346v1
• Date: Tue, 12 Mar 2024 06:04:50 GMT
• Title: Complementing Event Streams and RGB Frames for Hand Mesh Reconstruction
• Authors: Jianping Jiang, Xinyu Zhou, Bingxuan Wang, Xiaoming Deng, Chao Xu, Boxin Shi
• Abstract summary: We propose EvRGBHand -- the first approach for 3D hand mesh reconstruction with an event camera and an RGB camera compensating for each other. EvRGBHand can tackle overexposure and motion blur issues in RGB-based HMR and foreground scarcity and background overflow issues in event-based HMR.
• Score: 51.87279764576998
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reliable hand mesh reconstruction (HMR) from commonly-used color and depth sensors is challenging especially under scenarios with varied illuminations and fast motions. Event camera is a highly promising alternative for its high dynamic range and dense temporal resolution properties, but it lacks key texture appearance for hand mesh reconstruction. In this paper, we propose EvRGBHand -- the first approach for 3D hand mesh reconstruction with an event camera and an RGB camera compensating for each other. By fusing two modalities of data across time, space, and information dimensions,EvRGBHand can tackle overexposure and motion blur issues in RGB-based HMR and foreground scarcity and background overflow issues in event-based HMR. We further propose EvRGBDegrader, which allows our model to generalize effectively in challenging scenes, even when trained solely on standard scenes, thus reducing data acquisition costs. Experiments on real-world data demonstrate that EvRGBHand can effectively solve the challenging issues when using either type of camera alone via retaining the merits of both, and shows the potential of generalization to outdoor scenes and another type of event camera.

Related papers

• E-3DGS: Gaussian Splatting with Exposure and Motion Events [29.042018288378447]
  We propose E-3DGS, a novel event-based approach that partitions events into motion and exposure. We introduce a novel integration of 3DGS with exposure events for high-quality reconstruction of explicit scene representations. Our method is faster and delivers better reconstruction quality than event-based NeRF while being more cost-effective than NeRF methods.
  arXiv  Detail & Related papers  (2024-10-22T13:17:20Z)
• EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting [76.02450110026747]
  Event cameras, inspired by biological vision, record pixel-wise intensity changes asynchronously with high temporal resolution. We propose Event-Aided Free-Trajectory 3DGS, which seamlessly integrates the advantages of event cameras into 3DGS. We evaluate our method on the public Tanks and Temples benchmark and a newly collected real-world dataset, RealEv-DAVIS.
  arXiv  Detail & Related papers  (2024-10-20T13:44:24Z)
• SpikeNVS: Enhancing Novel View Synthesis from Blurry Images via Spike Camera [78.20482568602993]
  Conventional RGB cameras are susceptible to motion blur. Neuromorphic cameras like event and spike cameras inherently capture more comprehensive temporal information. Our design can enhance novel view synthesis across NeRF and 3DGS.
  arXiv  Detail & Related papers  (2024-04-10T03:31:32Z)
• EvPlug: Learn a Plug-and-Play Module for Event and Image Fusion [55.367269556557645]
  EvPlug learns a plug-and-play event and image fusion module from the supervision of the existing RGB-based model. We demonstrate the superiority of EvPlug in several vision tasks such as object detection, semantic segmentation, and 3D hand pose estimation.
  arXiv  Detail & Related papers  (2023-12-28T10:05:13Z)
• 3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera [59.846927201816776]
  This paper introduces the first framework for 3D tracking of two fast-moving and interacting hands from a single monocular event camera. Our approach tackles the left-right hand ambiguity with a novel semi-supervised feature-wise attention mechanism and integrates an intersection loss to fix hand collisions.
  arXiv  Detail & Related papers  (2023-12-21T18:59:57Z)
• HDR Reconstruction from Bracketed Exposures and Events [12.565039752529797]
  Reconstruction of high-quality HDR images is at the core of modern computational photography. We present a multi-modal end-to-end learning-based HDR imaging system that fuses bracketed images and event in the feature domain. Our framework exploits the higher temporal resolution of events by sub-sampling the input event streams using a sliding window.
  arXiv  Detail & Related papers  (2022-03-28T15:04:41Z)
• RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video [76.86512780916827]
  We present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera. In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN. We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline.
  arXiv  Detail & Related papers  (2021-06-22T12:53:56Z)
• RGB-D-E: Event Camera Calibration for Fast 6-DOF Object Tracking [16.06615504110132]
  We propose to use an event-based camera to increase the speed of 3D object tracking in 6 degrees of freedom. This application requires handling very high object speed to convey compelling AR experiences. We develop a deep learning approach, which combines an existing RGB-D network along with a novel event-based network in a cascade fashion.
  arXiv  Detail & Related papers  (2020-06-09T01:55:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.