MonoHuman: Animatable Human Neural Field from Monocular Video
- URL: http://arxiv.org/abs/2304.02001v1
- Date: Tue, 4 Apr 2023 17:55:03 GMT
- Title: MonoHuman: Animatable Human Neural Field from Monocular Video
- Authors: Zhengming Yu, Wei Cheng, Xian Liu, Wayne Wu, Kwan-Yee Lin
- Abstract summary: We propose a novel framework MonoHuman, which robustly renders view-consistent and high-fidelity avatars under arbitrary novel poses.
Our key insight is to model the deformation field with bi-directional constraints and explicitly leverage the off-the-peg information to reason the feature for coherent results.
- Score: 30.113937856494726
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Animating virtual avatars with free-view control is crucial for various
applications like virtual reality and digital entertainment. Previous studies
have attempted to utilize the representation power of the neural radiance field
(NeRF) to reconstruct the human body from monocular videos. Recent works
propose to graft a deformation network into the NeRF to further model the
dynamics of the human neural field for animating vivid human motions. However,
such pipelines either rely on pose-dependent representations or fall short of
motion coherency due to frame-independent optimization, making it difficult to
generalize to unseen pose sequences realistically. In this paper, we propose a
novel framework MonoHuman, which robustly renders view-consistent and
high-fidelity avatars under arbitrary novel poses. Our key insight is to model
the deformation field with bi-directional constraints and explicitly leverage
the off-the-peg keyframe information to reason the feature correlations for
coherent results. Specifically, we first propose a Shared Bidirectional
Deformation module, which creates a pose-independent generalizable deformation
field by disentangling backward and forward deformation correspondences into
shared skeletal motion weight and separate non-rigid motions. Then, we devise a
Forward Correspondence Search module, which queries the correspondence feature
of keyframes to guide the rendering network. The rendered results are thus
multi-view consistent with high fidelity, even under challenging novel pose
settings. Extensive experiments demonstrate the superiority of our proposed
MonoHuman over state-of-the-art methods.
Related papers
- Motion-Oriented Compositional Neural Radiance Fields for Monocular Dynamic Human Modeling [10.914612535745789]
This paper introduces Motion-oriented Compositional Neural Radiance Fields (MoCo-NeRF)
MoCo-NeRF is a framework designed to perform free-viewpoint rendering of monocular human videos.
arXiv Detail & Related papers (2024-07-16T17:59:01Z) - HR Human: Modeling Human Avatars with Triangular Mesh and High-Resolution Textures from Videos [52.23323966700072]
We present a framework for acquiring human avatars that are attached with high-resolution physically-based material textures and mesh from monocular video.
Our method introduces a novel information fusion strategy to combine the information from the monocular video and synthesize virtual multi-view images.
Experiments show that our approach outperforms previous representations in terms of high fidelity, and this explicit result supports deployment on common triangulars.
arXiv Detail & Related papers (2024-05-18T11:49:09Z) - Instant-NVR: Instant Neural Volumetric Rendering for Human-object
Interactions from Monocular RGBD Stream [14.844982083586306]
We propose Instant-NVR, a neural approach for instant volumetric human-object tracking and rendering using a single RGBD camera.
In the tracking front-end, we adopt a robust human-object capture scheme to provide sufficient motion priors.
We also provide an on-the-fly reconstruction scheme of the dynamic/static radiance fields via efficient motion-prior searching.
arXiv Detail & Related papers (2023-04-06T16:09:51Z) - GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from
Multi-view Images [79.39247661907397]
We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images.
Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
arXiv Detail & Related papers (2023-03-24T03:32:02Z) - Drivable Volumetric Avatars using Texel-Aligned Features [52.89305658071045]
Photo telepresence requires both high-fidelity body modeling and faithful driving to enable dynamically synthesized appearance.
We propose an end-to-end framework that addresses two core challenges in modeling and driving full-body avatars of real people.
arXiv Detail & Related papers (2022-07-20T09:28:16Z) - Neural Rendering of Humans in Novel View and Pose from Monocular Video [68.37767099240236]
We introduce a new method that generates photo-realistic humans under novel views and poses given a monocular video as input.
Our method significantly outperforms existing approaches under unseen poses and novel views given monocular videos as input.
arXiv Detail & Related papers (2022-04-04T03:09:20Z) - Learning Motion-Dependent Appearance for High-Fidelity Rendering of
Dynamic Humans from a Single Camera [49.357174195542854]
A key challenge of learning the dynamics of the appearance lies in the requirement of a prohibitively large amount of observations.
We show that our method can generate a temporally coherent video of dynamic humans for unseen body poses and novel views given a single view video.
arXiv Detail & Related papers (2022-03-24T00:22:03Z) - Animatable Neural Radiance Fields from Monocular RGB Video [72.6101766407013]
We present animatable neural radiance fields for detailed human avatar creation from monocular videos.
Our approach extends neural radiance fields to the dynamic scenes with human movements via introducing explicit pose-guided deformation.
In experiments we show that the proposed approach achieves 1) implicit human geometry and appearance reconstruction with high-quality details, 2) photo-realistic rendering of the human from arbitrary views, and 3) animation of the human with arbitrary poses.
arXiv Detail & Related papers (2021-06-25T13:32:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.