NeuMan: Neural Human Radiance Field from a Single Video

• URL: http://arxiv.org/abs/2203.12575v1
• Date: Wed, 23 Mar 2022 17:35:50 GMT
• Title: NeuMan: Neural Human Radiance Field from a Single Video
• Authors: Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, Anurag Ranjan
• Abstract summary: We train two NeRF models: a human NeRF model and a scene NeRF model. Our method is able to learn subject specific details, including cloth wrinkles and accessories, from just a 10 seconds video clip.
• Score: 26.7471970027198
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Photorealistic rendering and reposing of humans is important for enabling augmented reality experiences. We propose a novel framework to reconstruct the human and the scene that can be rendered with novel human poses and views from just a single in-the-wild video. Given a video captured by a moving camera, we train two NeRF models: a human NeRF model and a scene NeRF model. To train these models, we rely on existing methods to estimate the rough geometry of the human and the scene. Those rough geometry estimates allow us to create a warping field from the observation space to the canonical pose-independent space, where we train the human model in. Our method is able to learn subject specific details, including cloth wrinkles and accessories, from just a 10 seconds video clip, and to provide high quality renderings of the human under novel poses, from novel views, together with the background.

Related papers

• Human Gaussian Splatting: Real-time Rendering of Animatable Avatars [8.719797382786464]
  This work addresses the problem of real-time rendering of photorealistic human body avatars learned from multi-view videos. We propose an animatable human model based on 3D Gaussian Splatting, that has recently emerged as a very efficient alternative to neural radiance fields. Our method achieves 1.5 dB PSNR improvement over the state-of-the-art on THuman4 dataset while being able to render in real-time (80 fps for 512x512 resolution)
  arXiv  Detail & Related papers  (2023-11-28T12:05:41Z)
• PixelHuman: Animatable Neural Radiance Fields from Few Images [27.932366091437103]
  We propose PixelHuman, a novel rendering model that generates animatable human scenes from a few images of a person. Our method differs from existing methods in that it can generalize to any input image for animatable human synthesis. Our experiments show that our method achieves state-of-the-art performance in multiview and novel pose synthesis from few-shot images.
  arXiv  Detail & Related papers  (2023-07-18T08:41:17Z)
• Putting People in Their Place: Affordance-Aware Human Insertion into Scenes [61.63825003487104]
  We study the problem of inferring scene affordances by presenting a method for realistically inserting people into scenes. Given a scene image with a marked region and an image of a person, we insert the person into the scene while respecting the scene affordances. Our model can infer the set of realistic poses given the scene context, re-pose the reference person, and harmonize the composition.
  arXiv  Detail & Related papers  (2023-04-27T17:59:58Z)
• ActorsNeRF: Animatable Few-shot Human Rendering with Generalizable NeRFs [61.677180970486546]
  We propose a novel animatable NeRF called ActorsNeRF. It is first pre-trained on diverse human subjects, and then adapted with few-shot monocular video frames for a new actor with unseen poses. We quantitatively and qualitatively demonstrate that ActorsNeRF significantly outperforms the existing state-of-the-art on few-shot generalization to new people and poses on multiple datasets.
  arXiv  Detail & Related papers  (2023-04-27T17:58:48Z)
• Compositional 3D Human-Object Neural Animation [93.38239238988719]
  Human-object interactions (HOIs) are crucial for human-centric scene understanding applications such as human-centric visual generation, AR/VR, and robotics. In this paper, we address this challenge in HOI animation from a compositional perspective. We adopt neural human-object deformation to model and render HOI dynamics based on implicit neural representations.
  arXiv  Detail & Related papers  (2023-04-27T10:04:56Z)
• Vid2Avatar: 3D Avatar Reconstruction from Videos in the Wild via Self-supervised Scene Decomposition [40.46674919612935]
  We present Vid2Avatar, a method to learn human avatars from monocular in-the-wild videos. Our method does not require any groundtruth supervision or priors extracted from large datasets of clothed human scans. It solves the tasks of scene decomposition and surface reconstruction directly in 3D by modeling both the human and the background in the scene jointly.
  arXiv  Detail & Related papers  (2023-02-22T18:59:17Z)
• Neural Novel Actor: Learning a Generalized Animatable Neural Representation for Human Actors [98.24047528960406]
  We propose a new method for learning a generalized animatable neural representation from a sparse set of multi-view imagery of multiple persons. The learned representation can be used to synthesize novel view images of an arbitrary person from a sparse set of cameras, and further animate them with the user's pose control.
  arXiv  Detail & Related papers  (2022-08-25T07:36:46Z)
• H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion [42.4185273307021]
  We present H-NeRF, neural radiance fields for rendering and temporal (4D) reconstruction of a human in motion as captured by a sparse set of cameras or even from a monocular video. Our NeRF-inspired approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations.
  arXiv  Detail & Related papers  (2021-10-26T14:51:36Z)
• Animatable Neural Radiance Fields from Monocular RGB Video [72.6101766407013]
  We present animatable neural radiance fields for detailed human avatar creation from monocular videos. Our approach extends neural radiance fields to the dynamic scenes with human movements via introducing explicit pose-guided deformation. In experiments we show that the proposed approach achieves 1) implicit human geometry and appearance reconstruction with high-quality details, 2) photo-realistic rendering of the human from arbitrary views, and 3) animation of the human with arbitrary poses.
  arXiv  Detail & Related papers  (2021-06-25T13:32:23Z)
• Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control [80.79820002330457]
  We propose a new method for high-quality synthesis of humans from arbitrary viewpoints and under arbitrary controllable poses. Our method achieves better quality than the state-of-the-arts on playback as well as novel pose synthesis, and can even generalize well to new poses that starkly differ from the training poses.
  arXiv  Detail & Related papers  (2021-06-03T17:40:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.