Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control

• URL: http://arxiv.org/abs/2106.02019v1
• Date: Thu, 3 Jun 2021 17:40:48 GMT
• Title: Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control
• Authors: Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, Christian Theobalt
• Abstract summary: We propose a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses.
• Score: 80.79820002330457
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose Neural Actor (NA), a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method is built upon recent neural scene representation and rendering works which learn representations of geometry and appearance from only 2D images. While existing works demonstrated compelling rendering of static scenes and playback of dynamic scenes, photo-realistic reconstruction and rendering of humans with neural implicit methods, in particular under user-controlled novel poses, is still difficult. To address this problem, we utilize a coarse body model as the proxy to unwarp the surrounding 3D space into a canonical pose. A neural radiance field learns pose-dependent geometric deformations and pose- and view-dependent appearance effects in the canonical space from multi-view video input. To synthesize novel views of high fidelity dynamic geometry and appearance, we leverage 2D texture maps defined on the body model as latent variables for predicting residual deformations and the dynamic appearance. Experiments demonstrate that our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses. Furthermore, our method also supports body shape control of the synthesized results.

Related papers

• Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors [98.24047528960406]
  We propose a new method for learning a generalized animatable neural representation from a sparse set of multi-view imagery of multiple persons. The learned representation can be used to synthesize novel view images of an arbitrary person from a sparse set of cameras, and further animate them with the user's pose control.
  arXiv  Detail & Related papers  (2022-08-25T07:36:46Z)
• DANBO: Disentangled Articulated Neural Body Representations via Graph Neural Networks [12.132886846993108]
  High-resolution models enable photo-realistic avatars but at the cost of requiring studio settings not available to end users. Our goal is to create avatars directly from raw images without relying on expensive studio setups and surface tracking. We introduce a three-stage method that induces two inductive biases to better disentangled pose-dependent deformation.
  arXiv  Detail & Related papers  (2022-05-03T17:56:46Z)
• Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis [52.720314035084215]
  This work targets at using a general deep learning framework to synthesize free-viewpoint images of arbitrary human performers. We present a simple yet powerful framework, named Generalizable Neural Performer (GNR), that learns a generalizable and robust neural body representation. Experiments on GeneBody-1.0 and ZJU-Mocap show better robustness of our methods than recent state-of-the-art generalizable methods.
  arXiv  Detail & Related papers  (2022-04-25T17:14:22Z)
• Neural Rendering of Humans in Novel View and Pose from Monocular Video [68.37767099240236]
  We introduce a new method that generates photo-realistic humans under novel views and poses given a monocular video as input. Our method significantly outperforms existing approaches under unseen poses and novel views given monocular videos as input.
  arXiv  Detail & Related papers  (2022-04-04T03:09:20Z)
• LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies [78.17425779503047]
  We propose a novel neural implicit representation for the human body. It is fully differentiable and optimizable with disentangled shape and pose latent spaces. Our model can be trained and fine-tuned directly on non-watertight raw data with well-designed losses.
  arXiv  Detail & Related papers  (2021-11-30T04:10:57Z)
• Animatable Neural Radiance Fields from Monocular RGB Video [72.6101766407013]
  We present animatable neural radiance fields for detailed human avatar creation from monocular videos. Our approach extends neural radiance fields to the dynamic scenes with human movements via introducing explicit pose-guided deformation. In experiments we show that the proposed approach achieves 1) implicit human geometry and appearance reconstruction with high-quality details, 2) photo-realistic rendering of the human from arbitrary views, and 3) animation of the human with arbitrary poses.
  arXiv  Detail & Related papers  (2021-06-25T13:32:23Z)
• Learning Compositional Radiance Fields of Dynamic Human Heads [13.272666180264485]
  We propose a novel compositional 3D representation that combines the best of previous methods to produce both higher-resolution and faster results. Differentiable volume rendering is employed to compute photo-realistic novel views of the human head and upper body. Our approach achieves state-of-the-art results for synthesizing novel views of dynamic human heads and the upper body.
  arXiv  Detail & Related papers  (2020-12-17T22:19:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.