Surface-Aligned Neural Radiance Fields for Controllable 3D Human Synthesis

• URL: http://arxiv.org/abs/2201.01683v1
• Date: Wed, 5 Jan 2022 16:25:32 GMT
• Title: Surface-Aligned Neural Radiance Fields for Controllable 3D Human Synthesis
• Authors: Tianhan Xu, Yasuhiro Fujita, Eiichi Matsumoto
• Abstract summary: We propose a new method for reconstructing implicit 3D human models from sparse multi-view RGB videos. Our method defines the neural scene representation on the mesh surface points and signed distances from the surface of a human body mesh.
• Score: 4.597864989500202
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a new method for reconstructing controllable implicit 3D human models from sparse multi-view RGB videos. Our method defines the neural scene representation on the mesh surface points and signed distances from the surface of a human body mesh. We identify an indistinguishability issue that arises when a point in 3D space is mapped to its nearest surface point on a mesh for learning surface-aligned neural scene representation. To address this issue, we propose projecting a point onto a mesh surface using a barycentric interpolation with modified vertex normals. Experiments with the ZJU-MoCap and Human3.6M datasets show that our approach achieves a higher quality in a novel-view and novel-pose synthesis than existing methods. We also demonstrate that our method easily supports the control of body shape and clothes.

Related papers

• DreamMesh4D: Video-to-4D Generation with Sparse-Controlled Gaussian-Mesh Hybrid Representation [10.250715657201363]
  We introduce DreamMesh4D, a novel framework combining mesh representation with geometric skinning technique to generate high-quality 4D object from a monocular video. Our method is compatible with modern graphic pipelines, showcasing its potential in the 3D gaming and film industry.
  arXiv  Detail & Related papers  (2024-10-09T10:41:08Z)
• Total-Decom: Decomposed 3D Scene Reconstruction with Minimal Interaction [51.3632308129838]
  We present Total-Decom, a novel method for decomposed 3D reconstruction with minimal human interaction. Our approach seamlessly integrates the Segment Anything Model (SAM) with hybrid implicit-explicit neural surface representations and a mesh-based region-growing technique for accurate 3D object decomposition. We extensively evaluate our method on benchmark datasets and demonstrate its potential for downstream applications, such as animation and scene editing.
  arXiv  Detail & Related papers  (2024-03-28T11:12:33Z)
• DiViNeT: 3D Reconstruction from Disparate Views via Neural Template Regularization [7.488962492863031]
  We present a volume rendering-based neural surface reconstruction method that takes as few as three disparate RGB images as input. Our key idea is to regularize the reconstruction, which is severely ill-posed and leaving significant gaps between the sparse views. Our approach achieves the best reconstruction quality among existing methods in the presence of such sparse views.
  arXiv  Detail & Related papers  (2023-06-07T18:05:14Z)
• Sampling is Matter: Point-guided 3D Human Mesh Reconstruction [0.0]
  This paper presents a simple yet powerful method for 3D human mesh reconstruction from a single RGB image. Experimental results on benchmark datasets show that the proposed method efficiently improves the performance of 3D human mesh reconstruction.
  arXiv  Detail & Related papers  (2023-04-19T08:45:26Z)
• NeAT: Learning Neural Implicit Surfaces with Arbitrary Topologies from Multi-view Images [17.637064969966847]
  NeAT is a new neural rendering framework that learns implicit surfaces with arbitrary topologies from multi-view images. NeAT supports easy field-to-mesh conversion using the classic Marching Cubes algorithm. Our approach is able to faithfully reconstruct both watertight and non-watertight surfaces.
  arXiv  Detail & Related papers  (2023-03-21T16:49:41Z)
• Neural Capture of Animatable 3D Human from Monocular Video [38.974181971541846]
  We present a novel paradigm of building an animatable 3D human representation from a monocular video input, such that it can be rendered in any unseen poses and views. Our method is based on a dynamic Neural Radiance Field (NeRF) rigged by a mesh-based parametric 3D human model serving as a geometry proxy.
  arXiv  Detail & Related papers  (2022-08-18T09:20:48Z)
• Neural Volumetric Object Selection [126.04480613166194]
  We introduce an approach for selecting objects in neural volumetric 3D representations, such as multi-plane images (MPI) and neural radiance fields (NeRF) Our approach takes a set of foreground and background 2D user scribbles in one view and automatically estimates a 3D segmentation of the desired object, which can be rendered into novel views.
  arXiv  Detail & Related papers  (2022-05-30T08:55:20Z)
• N-Cloth: Predicting 3D Cloth Deformation with Mesh-Based Networks [69.94313958962165]
  We present a novel mesh-based learning approach (N-Cloth) for plausible 3D cloth deformation prediction. We use graph convolution to transform the cloth and object meshes into a latent space to reduce the non-linearity in the mesh space. Our approach can handle complex cloth meshes with up to $100$K triangles and scenes with various objects corresponding to SMPL humans, Non-SMPL humans, or rigid bodies.
  arXiv  Detail & Related papers  (2021-12-13T03:13:11Z)
• KAMA: 3D Keypoint Aware Body Mesh Articulation [79.04090630502782]
  We propose an analytical solution to articulate a parametric body model, SMPL, via a set of straightforward geometric transformations. Our approach offers significantly better alignment to image content when compared to state-of-the-art approaches. Results on the challenging 3DPW and Human3.6M demonstrate that our approach yields state-of-the-art body mesh fittings.
  arXiv  Detail & Related papers  (2021-04-27T23:01:03Z)
• Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose [70.23652933572647]
  We propose a novel graph convolutional neural network (GraphCNN)-based system that estimates the 3D coordinates of human mesh vertices directly from the 2D human pose. We show that our Pose2Mesh outperforms the previous 3D human pose and mesh estimation methods on various benchmark datasets.
  arXiv  Detail & Related papers  (2020-08-20T16:01:56Z)
• Pix2Surf: Learning Parametric 3D Surface Models of Objects from Images [64.53227129573293]
  We investigate the problem of learning to generate 3D parametric surface representations for novel object instances, as seen from one or more views. We design neural networks capable of generating high-quality parametric 3D surfaces which are consistent between views. Our method is supervised and trained on a public dataset of shapes from common object categories.
  arXiv  Detail & Related papers  (2020-08-18T06:33:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.