Neural Image-based Avatars: Generalizable Radiance Fields for Human Avatar Modeling

• URL: http://arxiv.org/abs/2304.04897v1
• Date: Mon, 10 Apr 2023 23:53:28 GMT
• Title: Neural Image-based Avatars: Generalizable Radiance Fields for Human Avatar Modeling
• Authors: Youngjoong Kwon, Dahun Kim, Duygu Ceylan, Henry Fuchs
• Abstract summary: We present a method that enables novel views and novel poses of arbitrary human performers from sparse multi-view images. A key ingredient of our method is a hybrid appearance blending module that combines the advantages of the implicit body NeRF representation and image-based rendering.
• Score: 28.242591786838936
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a method that enables synthesizing novel views and novel poses of arbitrary human performers from sparse multi-view images. A key ingredient of our method is a hybrid appearance blending module that combines the advantages of the implicit body NeRF representation and image-based rendering. Existing generalizable human NeRF methods that are conditioned on the body model have shown robustness against the geometric variation of arbitrary human performers. Yet they often exhibit blurry results when generalized onto unseen identities. Meanwhile, image-based rendering shows high-quality results when sufficient observations are available, whereas it suffers artifacts in sparse-view settings. We propose Neural Image-based Avatars (NIA) that exploits the best of those two methods: to maintain robustness under new articulations and self-occlusions while directly leveraging the available (sparse) source view colors to preserve appearance details of new subject identities. Our hybrid design outperforms recent methods on both in-domain identity generalization as well as challenging cross-dataset generalization settings. Also, in terms of the pose generalization, our method outperforms even the per-subject optimized animatable NeRF methods. The video results are available at https://youngjoongunc.github.io/nia

Related papers

• HR Human: Modeling Human Avatars with Triangular Mesh and High-Resolution Textures from Videos [52.23323966700072]
  We present a framework for acquiring human avatars that are attached with high-resolution physically-based material textures and mesh from monocular video. Our method introduces a novel information fusion strategy to combine the information from the monocular video and synthesize virtual multi-view images. Experiments show that our approach outperforms previous representations in terms of high fidelity, and this explicit result supports deployment on common triangulars.
  arXiv  Detail & Related papers  (2024-05-18T11:49:09Z)
• Hybrid Neural Rendering for Large-Scale Scenes with Motion Blur [68.24599239479326]
  We develop a hybrid neural rendering model that makes image-based representation and neural 3D representation join forces to render high-quality, view-consistent images. Our model surpasses state-of-the-art point-based methods for novel view synthesis.
  arXiv  Detail & Related papers  (2023-04-25T08:36:33Z)
• HQ3DAvatar: High Quality Controllable 3D Head Avatar [65.70885416855782]
  This paper presents a novel approach to building highly photorealistic digital head avatars. Our method learns a canonical space via an implicit function parameterized by a neural network. At test time, our method is driven by a monocular RGB video.
  arXiv  Detail & Related papers  (2023-03-25T13:56:33Z)
• GM-NeRF: Learning Generalizable Model-based Neural Radiance Fields from Multi-view Images [79.39247661907397]
  We introduce an effective framework Generalizable Model-based Neural Radiance Fields to synthesize free-viewpoint images. Specifically, we propose a geometry-guided attention mechanism to register the appearance code from multi-view 2D images to a geometry proxy.
  arXiv  Detail & Related papers  (2023-03-24T03:32:02Z)
• DINAR: Diffusion Inpainting of Neural Textures for One-Shot Human Avatars [7.777410338143783]
  We present an approach for creating realistic rigged fullbody avatars from single RGB images. Our method uses neural textures combined with the SMPL-X body model to achieve photo-realistic quality of avatars. In the experiments, our approach achieves state-of-the-art rendering quality and good generalization to new poses and viewpoints.
  arXiv  Detail & Related papers  (2023-03-16T15:04:10Z)
• ARAH: Animatable Volume Rendering of Articulated Human SDFs [37.48271522183636]
  We propose a model to create animatable clothed human avatars with detailed geometry that generalize well to out-of-distribution poses. Our algorithm enables efficient point sampling and accurate point canonicalization while generalizing well to unseen poses. Our method achieves state-of-the-art performance on geometry and appearance reconstruction while creating animatable avatars.
  arXiv  Detail & Related papers  (2022-10-18T17:56:59Z)
• Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis [52.720314035084215]
  This work targets at using a general deep learning framework to synthesize free-viewpoint images of arbitrary human performers. We present a simple yet powerful framework, named Generalizable Neural Performer (GNR), that learns a generalizable and robust neural body representation. Experiments on GeneBody-1.0 and ZJU-Mocap show better robustness of our methods than recent state-of-the-art generalizable methods.
  arXiv  Detail & Related papers  (2022-04-25T17:14:22Z)
• MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images [32.84481902544513]
  This paper deals with rendering novel views and novel poses for a person unseen in training, using only multiview images as input. Key ingredient is a dedicated representation combining a canonical NeRF and a volume deformation scheme. Experiments on both real and synthetic data with the novel view synthesis and pose animation tasks collectively demonstrate the efficacy of our method.
  arXiv  Detail & Related papers  (2022-03-31T08:09:03Z)
• Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering [34.80975358673563]
  We propose a novel approach that learns generalizable neural radiance fields based on a parametric human body model for robust performance capture. Experiments on the ZJU-MoCap and AIST datasets show that our method significantly outperforms recent generalizable NeRF methods on unseen identities and poses.
  arXiv  Detail & Related papers  (2021-09-15T17:32:46Z)
• Neural Re-Rendering of Humans from a Single Image [80.53438609047896]
  We propose a new method for neural re-rendering of a human under a novel user-defined pose and viewpoint. Our algorithm represents body pose and shape as a parametric mesh which can be reconstructed from a single image.
  arXiv  Detail & Related papers  (2021-01-11T18:53:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.