Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors

• URL: http://arxiv.org/abs/2208.11905v2
• Date: Tue, 23 May 2023 06:56:49 GMT
• Title: Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors
• Authors: Yiming Wang, Qingzhe Gao, Libin Liu, Lingjie Liu, Christian Theobalt, Baoquan Chen
• Abstract summary: We propose a new method for learning a generalized animatable neural representation from a sparse set of multi-view imagery of multiple persons. The learned representation can be used to synthesize novel view images of an arbitrary person from a sparse set of cameras, and further animate them with the user's pose control.
• Score: 98.24047528960406
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a new method for learning a generalized animatable neural human representation from a sparse set of multi-view imagery of multiple persons. The learned representation can be used to synthesize novel view images of an arbitrary person from a sparse set of cameras, and further animate them with the user's pose control. While existing methods can either generalize to new persons or synthesize animations with user control, none of them can achieve both at the same time. We attribute this accomplishment to the employment of a 3D proxy for a shared multi-person human model, and further the warping of the spaces of different poses to a shared canonical pose space, in which we learn a neural field and predict the person- and pose-dependent deformations, as well as appearance with the features extracted from input images. To cope with the complexity of the large variations in body shapes, poses, and clothing deformations, we design our neural human model with disentangled geometry and appearance. Furthermore, we utilize the image features both at the spatial point and on the surface points of the 3D proxy for predicting person- and pose-dependent properties. Experiments show that our method significantly outperforms the state-of-the-arts on both tasks. The video and code are available at https://talegqz.github.io/neural_novel_actor.

Related papers

• PixelHuman: Animatable Neural Radiance Fields from Few Images [27.932366091437103]
  We propose PixelHuman, a novel rendering model that generates animatable human scenes from a few images of a person. Our method differs from existing methods in that it can generalize to any input image for animatable human synthesis. Our experiments show that our method achieves state-of-the-art performance in multiview and novel pose synthesis from few-shot images.
  arXiv  Detail & Related papers  (2023-07-18T08:41:17Z)
• Compositional 3D Human-Object Neural Animation [93.38239238988719]
  Human-object interactions (HOIs) are crucial for human-centric scene understanding applications such as human-centric visual generation, AR/VR, and robotics. In this paper, we address this challenge in HOI animation from a compositional perspective. We adopt neural human-object deformation to model and render HOI dynamics based on implicit neural representations.
  arXiv  Detail & Related papers  (2023-04-27T10:04:56Z)
• Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis [52.720314035084215]
  This work targets at using a general deep learning framework to synthesize free-viewpoint images of arbitrary human performers. We present a simple yet powerful framework, named Generalizable Neural Performer (GNR), that learns a generalizable and robust neural body representation. Experiments on GeneBody-1.0 and ZJU-Mocap show better robustness of our methods than recent state-of-the-art generalizable methods.
  arXiv  Detail & Related papers  (2022-04-25T17:14:22Z)
• MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images [32.84481902544513]
  This paper deals with rendering novel views and novel poses for a person unseen in training, using only multiview images as input. Key ingredient is a dedicated representation combining a canonical NeRF and a volume deformation scheme. Experiments on both real and synthetic data with the novel view synthesis and pose animation tasks collectively demonstrate the efficacy of our method.
  arXiv  Detail & Related papers  (2022-03-31T08:09:03Z)
• Creating and Reenacting Controllable 3D Humans with Differentiable Rendering [3.079885946230076]
  This paper proposes a new end-to-end neural rendering architecture to transfer appearance and reenact human actors. Our method leverages a carefully designed graph convolutional network (GCN) to model the human body manifold structure. By taking advantages of both different synthesisiable rendering and the 3D parametric model, our method is fully controllable.
  arXiv  Detail & Related papers  (2021-10-22T12:40:09Z)
• Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothing [49.32522765356914]
  We learn to animate people in clothing as a function of the body pose. We learn to map every point in the space to a canonical space, where a learned deformation field is applied to model non-rigid effects. Neural-GIF can be trained on raw 3D scans and reconstructs detailed complex surface geometry and deformations.
  arXiv  Detail & Related papers  (2021-08-19T17:25:16Z)
• Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control [80.79820002330457]
  We propose a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses.
  arXiv  Detail & Related papers  (2021-06-03T17:40:48Z)
• S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling [103.65625425020129]
  We represent the pedestrian's shape, pose and skinning weights as neural implicit functions that are directly learned from data. We demonstrate the effectiveness of our approach on various datasets and show that our reconstructions outperform existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-01-17T02:16:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.