Complementing Event Streams and RGB Frames for Hand Mesh Reconstruction
- URL: http://arxiv.org/abs/2403.07346v1
- Date: Tue, 12 Mar 2024 06:04:50 GMT
- Title: Complementing Event Streams and RGB Frames for Hand Mesh Reconstruction
- Authors: Jianping Jiang, Xinyu Zhou, Bingxuan Wang, Xiaoming Deng, Chao Xu,
Boxin Shi
- Abstract summary: We propose EvRGBHand -- the first approach for 3D hand mesh reconstruction with an event camera and an RGB camera compensating for each other.
EvRGBHand can tackle overexposure and motion blur issues in RGB-based HMR and foreground scarcity and background overflow issues in event-based HMR.
- Score: 51.87279764576998
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Reliable hand mesh reconstruction (HMR) from commonly-used color and depth
sensors is challenging especially under scenarios with varied illuminations and
fast motions. Event camera is a highly promising alternative for its high
dynamic range and dense temporal resolution properties, but it lacks key
texture appearance for hand mesh reconstruction. In this paper, we propose
EvRGBHand -- the first approach for 3D hand mesh reconstruction with an event
camera and an RGB camera compensating for each other. By fusing two modalities
of data across time, space, and information dimensions,EvRGBHand can tackle
overexposure and motion blur issues in RGB-based HMR and foreground scarcity
and background overflow issues in event-based HMR. We further propose
EvRGBDegrader, which allows our model to generalize effectively in challenging
scenes, even when trained solely on standard scenes, thus reducing data
acquisition costs. Experiments on real-world data demonstrate that EvRGBHand
can effectively solve the challenging issues when using either type of camera
alone via retaining the merits of both, and shows the potential of
generalization to outdoor scenes and another type of event camera.
Related papers
- SpikeNVS: Enhancing Novel View Synthesis from Blurry Images via Spike Camera [78.20482568602993]
Conventional RGB cameras are susceptible to motion blur.
Neuromorphic cameras like event and spike cameras inherently capture more comprehensive temporal information.
Our design can enhance novel view synthesis across NeRF and 3DGS.
arXiv Detail & Related papers (2024-04-10T03:31:32Z) - EvPlug: Learn a Plug-and-Play Module for Event and Image Fusion [55.367269556557645]
EvPlug learns a plug-and-play event and image fusion module from the supervision of the existing RGB-based model.
We demonstrate the superiority of EvPlug in several vision tasks such as object detection, semantic segmentation, and 3D hand pose estimation.
arXiv Detail & Related papers (2023-12-28T10:05:13Z) - 3D Pose Estimation of Two Interacting Hands from a Monocular Event
Camera [59.846927201816776]
This paper introduces the first framework for 3D tracking of two fast-moving and interacting hands from a single monocular event camera.
Our approach tackles the left-right hand ambiguity with a novel semi-supervised feature-wise attention mechanism and integrates an intersection loss to fix hand collisions.
arXiv Detail & Related papers (2023-12-21T18:59:57Z) - EvHandPose: Event-based 3D Hand Pose Estimation with Sparse Supervision [50.060055525889915]
Event camera shows great potential in 3D hand pose estimation, especially addressing the challenges of fast motion and high dynamic range in a low-power way.
It is challenging to design event representation to encode hand motion information especially when the hands are not moving.
In this paper, we propose EvHandPose with novel hand flow representations in Event-to-Pose module for accurate hand pose estimation.
arXiv Detail & Related papers (2023-03-06T03:27:17Z) - HDR Reconstruction from Bracketed Exposures and Events [12.565039752529797]
Reconstruction of high-quality HDR images is at the core of modern computational photography.
We present a multi-modal end-to-end learning-based HDR imaging system that fuses bracketed images and event in the feature domain.
Our framework exploits the higher temporal resolution of events by sub-sampling the input event streams using a sliding window.
arXiv Detail & Related papers (2022-03-28T15:04:41Z) - E$^2$(GO)MOTION: Motion Augmented Event Stream for Egocentric Action
Recognition [21.199869051111367]
Event cameras capture pixel-level intensity changes in the form of "events"
N-EPIC-Kitchens is the first event-based camera extension of the large-scale EPIC-Kitchens dataset.
We show that event data provides a comparable performance to RGB and optical flow, yet without any additional flow computation at deploy time.
arXiv Detail & Related papers (2021-12-07T09:43:08Z) - RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB
Video [76.86512780916827]
We present the first real-time method for motion capture of skeletal pose and 3D surface geometry of hands from a single RGB camera.
In order to address the inherent depth ambiguities in RGB data, we propose a novel multi-task CNN.
We experimentally verify the individual components of our RGB two-hand tracking and 3D reconstruction pipeline.
arXiv Detail & Related papers (2021-06-22T12:53:56Z) - RGB-D-E: Event Camera Calibration for Fast 6-DOF Object Tracking [16.06615504110132]
We propose to use an event-based camera to increase the speed of 3D object tracking in 6 degrees of freedom.
This application requires handling very high object speed to convey compelling AR experiences.
We develop a deep learning approach, which combines an existing RGB-D network along with a novel event-based network in a cascade fashion.
arXiv Detail & Related papers (2020-06-09T01:55:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.