Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors

• URL: http://arxiv.org/abs/2105.01859v2
• Date: Thu, 6 May 2021 07:38:27 GMT
• Title: Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors
• Authors: Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, Yebin Liu
• Abstract summary: We propose a human volumetric capture method that combines temporal fusion and deep implicit functions. We propose dynamic sliding to fuse depth observations together with topology consistency.
• Score: 67.88097893304274
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human volumetric capture is a long-standing topic in computer vision and computer graphics. Although high-quality results can be achieved using sophisticated off-line systems, real-time human volumetric capture of complex scenarios, especially using light-weight setups, remains challenging. In this paper, we propose a human volumetric capture method that combines temporal volumetric fusion and deep implicit functions. To achieve high-quality and temporal-continuous reconstruction, we propose dynamic sliding fusion to fuse neighboring depth observations together with topology consistency. Moreover, for detailed and complete surface generation, we propose detail-preserving deep implicit functions for RGBD input which can not only preserve the geometric details on the depth inputs but also generate more plausible texturing results. Results and experiments show that our method outperforms existing methods in terms of view sparsity, generalization capacity, reconstruction quality, and run-time efficiency.

Related papers

• DNS SLAM: Dense Neural Semantic-Informed SLAM [92.39687553022605]
  DNS SLAM is a novel neural RGB-D semantic SLAM approach featuring a hybrid representation. Our method integrates multi-view geometry constraints with image-based feature extraction to improve appearance details. Our experimental results achieve state-of-the-art performance on both synthetic data and real-world data tracking.
  arXiv  Detail & Related papers  (2023-11-30T21:34:44Z)
• HyperINR: A Fast and Predictive Hypernetwork for Implicit Neural Representations via Knowledge Distillation [31.44962361819199]
  Implicit Neural Representations (INRs) have recently exhibited immense potential in the field of scientific visualization. In this paper, we introduce HyperINR, a novel hypernetwork architecture capable of directly predicting the weights for a compact INR. By harnessing an ensemble of multiresolution hash encoding units in unison, the resulting INR attains state-of-the-art inference performance.
  arXiv  Detail & Related papers  (2023-04-09T08:10:10Z)
• Learning Dynamic View Synthesis With Few RGBD Cameras [60.36357774688289]
  We propose to utilize RGBD cameras to synthesize free-viewpoint videos of dynamic indoor scenes. We generate point clouds from RGBD frames and then render them into free-viewpoint videos via a neural feature. We introduce a simple Regional Depth-Inpainting module that adaptively inpaints missing depth values to render complete novel views.
  arXiv  Detail & Related papers  (2022-04-22T03:17:35Z)
• Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction [91.43066633305662]
  We propose a novel multi-task and multi-modal filtered transformer (MMFT) network for RGB-D salient object detection (SOD) Specifically, we unify three complementary tasks: depth estimation, salient object detection and contour estimation. The multi-task mechanism promotes the model to learn the task-aware features from the auxiliary tasks. Experiments show that it not only significantly surpasses the depth-based RGB-D SOD methods on multiple datasets, but also precisely predicts a high-quality depth map and salient contour at the same time.
  arXiv  Detail & Related papers  (2022-03-09T17:20:18Z)
• NeuralFusion: Neural Volumetric Rendering under Human-object Interactions [46.70371238621842]
  We propose a neural approach for volumetric human-object capture and rendering using sparse consumer RGBD sensors. For geometry modeling, we propose a neural implicit inference scheme with non-rigid key-volume fusion. We also introduce a layer-wise human-object texture rendering scheme, which combines volumetric and image-based rendering in both spatial and temporal domains.
  arXiv  Detail & Related papers  (2022-02-25T17:10:07Z)
• Unsupervised Single-shot Depth Estimation using Perceptual Reconstruction [0.0]
  This study presents the most recent advances in the field of generative neural networks, leveraging them to perform fully unsupervised single-shot depth synthesis. Two generators for RGB-to-depth and depth-to-RGB transfer are implemented and simultaneously optimized using the Wasserstein-1 distance and a novel perceptual reconstruction term. The success observed in this study suggests the great potential for unsupervised single-shot depth estimation in real-world applications.
  arXiv  Detail & Related papers  (2022-01-28T15:11:34Z)
• Geometry-Guided Progressive NeRF for Generalizable and Efficient Neural Human Rendering [139.159534903657]
  We develop a generalizable and efficient Neural Radiance Field (NeRF) pipeline for high-fidelity free-viewpoint human body details. To better tackle self-occlusion, we devise a geometry-guided multi-view feature integration approach. For achieving higher rendering efficiency, we introduce a geometry-guided progressive rendering pipeline.
  arXiv  Detail & Related papers  (2021-12-08T14:42:10Z)
• Unpaired Single-Image Depth Synthesis with cycle-consistent Wasserstein GANs [1.0499611180329802]
  Real-time estimation of actual environment depth is an essential module for various autonomous system tasks. In this study, latest advancements in the field of generative neural networks are leveraged to fully unsupervised single-image depth synthesis.
  arXiv  Detail & Related papers  (2021-03-31T09:43:38Z)
• Accurate RGB-D Salient Object Detection via Collaborative Learning [101.82654054191443]
  RGB-D saliency detection shows impressive ability on some challenge scenarios. We propose a novel collaborative learning framework where edge, depth and saliency are leveraged in a more efficient way.
  arXiv  Detail & Related papers  (2020-07-23T04:33:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.