Mesh Strikes Back: Fast and Efficient Human Reconstruction from RGB videos

• URL: http://arxiv.org/abs/2303.08808v1
• Date: Wed, 15 Mar 2023 17:57:13 GMT
• Title: Mesh Strikes Back: Fast and Efficient Human Reconstruction from RGB videos
• Authors: Rohit Jena, Pratik Chaudhari, James Gee, Ganesh Iyer, Siddharth Choudhary, Brandon M. Smith
• Abstract summary: Many methods employ deferred rendering, NeRFs and implicit methods to represent clothed humans. We provide a counter viewpoint by optimizing a SMPL+D mesh and an efficient, multi-resolution texture representation. We show competitive novel view synthesis and improvements in novel pose synthesis compared to NeRF-based methods.
• Score: 15.746993448290175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human reconstruction and synthesis from monocular RGB videos is a challenging problem due to clothing, occlusion, texture discontinuities and sharpness, and framespecific pose changes. Many methods employ deferred rendering, NeRFs and implicit methods to represent clothed humans, on the premise that mesh-based representations cannot capture complex clothing and textures from RGB, silhouettes, and keypoints alone. We provide a counter viewpoint to this fundamental premise by optimizing a SMPL+D mesh and an efficient, multi-resolution texture representation using only RGB images, binary silhouettes and sparse 2D keypoints. Experimental results demonstrate that our approach is more capable of capturing geometric details compared to visual hull, mesh-based methods. We show competitive novel view synthesis and improvements in novel pose synthesis compared to NeRF-based methods, which introduce noticeable, unwanted artifacts. By restricting the solution space to the SMPL+D model combined with differentiable rendering, we obtain dramatic speedups in compute, training times (up to 24x) and inference times (up to 192x). Our method therefore can be used as is or as a fast initialization to NeRF-based methods.

Related papers

• Few-Shot Multi-Human Neural Rendering Using Geometry Constraints [8.819403814092865]
  We present a method for recovering the shape and radiance of a scene consisting of multiple people given solely a few images. Existing approaches using implicit neural representations have achieved impressive results that deliver accurate geometry and appearance. We propose a neural implicit reconstruction method that addresses the inherent challenges of this task through the following contributions.
  arXiv  Detail & Related papers  (2025-02-11T00:10:58Z)
• Real-time Free-view Human Rendering from Sparse-view RGB Videos using Double Unprojected Textures [87.80984588545589]
  Real-time free-view human rendering from sparse-view RGB inputs is a challenging task due to the sensor scarcity and the tight time budget. Recent methods leverage 2D CNNs operating in texture space to learn rendering primitives. We present Double Unprojected Textures, which at the core disentangles coarse geometric deformation estimation from appearance synthesis.
  arXiv  Detail & Related papers  (2024-12-17T18:57:38Z)
• NeRF-Texture: Synthesizing Neural Radiance Field Textures [77.24205024987414]
  We propose a novel texture synthesis method with Neural Radiance Fields (NeRF) to capture and synthesize textures from given multi-view images. In the proposed NeRF texture representation, a scene with fine geometric details is disentangled into the meso-structure textures and the underlying base shape. We can synthesize NeRF-based textures through patch matching of latent features.
  arXiv  Detail & Related papers  (2024-12-13T09:41:48Z)
• Hybrid Explicit Representation for Ultra-Realistic Head Avatars [55.829497543262214]
  We introduce a novel approach to creating ultra-realistic head avatars and rendering them in real-time. UV-mapped 3D mesh is utilized to capture sharp and rich textures on smooth surfaces, while 3D Gaussian Splatting is employed to represent complex geometric structures. Experiments that our modeled results exceed those of state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-03-18T04:01:26Z)
• ConTex-Human: Free-View Rendering of Human from a Single Image with Texture-Consistent Synthesis [49.28239918969784]
  We introduce a texture-consistent back view synthesis module that could transfer the reference image content to the back view. We also propose a visibility-aware patch consistency regularization for texture mapping and refinement combined with the synthesized back view texture.
  arXiv  Detail & Related papers  (2023-11-28T13:55:53Z)
• Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives [70.32817882783608]
  We present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. Unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points.
  arXiv  Detail & Related papers  (2023-07-11T17:58:31Z)
• FastHuman: Reconstructing High-Quality Clothed Human in Minutes [18.643091757385626]
  We propose an approach for optimizing high-quality clothed human body shapes in minutes. Our method uses a mesh-based patch warping technique to ensure multi-view photometric consistency. Our approach has demonstrated promising results on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-11-26T05:16:39Z)
• View Synthesis with Sculpted Neural Points [64.40344086212279]
  Implicit neural representations have achieved impressive visual quality but have drawbacks in computational efficiency. We propose a new approach that performs view synthesis using point clouds. It is the first point-based method to achieve better visual quality than NeRF while being more than 100x faster in rendering speed.
  arXiv  Detail & Related papers  (2022-05-12T03:54:35Z)
• Deblur-NeRF: Neural Radiance Fields from Blurry Images [30.709331199256376]
  We propose De-NeRF, the first method that can recover a sharp NeRF from blurry input. We adopt an analysis-by-blur approach that reconstructs blurry views by simulating the blurring process. We demonstrate that our method can be used on both camera motion blur and defocus blur: the two most common types of blur in real scenes.
  arXiv  Detail & Related papers  (2021-11-29T01:49:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.