UVA: Towards Unified Volumetric Avatar for View Synthesis, Pose rendering, Geometry and Texture Editing

• URL: http://arxiv.org/abs/2304.06969v1
• Date: Fri, 14 Apr 2023 07:39:49 GMT
• Title: UVA: Towards Unified Volumetric Avatar for View Synthesis, Pose rendering, Geometry and Texture Editing
• Authors: Jinlong Fan and Jing Zhang and Dacheng Tao
• Abstract summary: We propose a new approach named textbfUnified textbfVolumetric textbfAvatar (textbfUVA) that enables local editing of both geometry and texture. UVA transforms each observation point to a canonical space using a skinning motion field and represents geometry and texture in separate neural fields. Experiments on multiple human avatars demonstrate that our UVA achieves novel view synthesis and novel pose rendering.
• Score: 83.0396740127043
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural radiance field (NeRF) has become a popular 3D representation method for human avatar reconstruction due to its high-quality rendering capabilities, e.g., regarding novel views and poses. However, previous methods for editing the geometry and appearance of the avatar only allow for global editing through body shape parameters and 2D texture maps. In this paper, we propose a new approach named \textbf{U}nified \textbf{V}olumetric \textbf{A}vatar (\textbf{UVA}) that enables local and independent editing of both geometry and texture, while retaining the ability to render novel views and poses. UVA transforms each observation point to a canonical space using a skinning motion field and represents geometry and texture in separate neural fields. Each field is composed of a set of structured latent codes that are attached to anchor nodes on a deformable mesh in canonical space and diffused into the entire space via interpolation, allowing for local editing. To address spatial ambiguity in code interpolation, we use a local signed height indicator. We also replace the view-dependent radiance color with a pose-dependent shading factor to better represent surface illumination in different poses. Experiments on multiple human avatars demonstrate that our UVA achieves competitive results in novel view synthesis and novel pose rendering while enabling local and independent editing of geometry and appearance. The source code will be released.

Related papers

• NECA: Neural Customizable Human Avatar [36.69012172745299]
  We introduce NECA, an approach capable of learning versatile human representation from monocular or sparse-view videos. The core of our approach is to represent humans in complementary dual spaces and predict disentangled neural fields of geometry, albedo, shadow, as well as an external lighting.
  arXiv  Detail & Related papers  (2024-03-15T14:23:06Z)
• Learning Naturally Aggregated Appearance for Efficient 3D Editing [94.47518916521065]
  We propose to replace the color field with an explicit 2D appearance aggregation, also called canonical image. To avoid the distortion effect and facilitate convenient editing, we complement the canonical image with a projection field that maps 3D points onto 2D pixels for texture lookup. Our representation, dubbed AGAP, well supports various ways of 3D editing (e.g., stylization, interactive drawing, and content extraction) with no need of re-optimization.
  arXiv  Detail & Related papers  (2023-12-11T18:59:31Z)
• Animating NeRFs from Texture Space: A Framework for Pose-Dependent Rendering of Human Performances [11.604386285817302]
  We introduce a novel NeRF-based framework for pose-dependent rendering of human performances. Our approach results in high-quality renderings for novel-view and novel-pose synthesis.
  arXiv  Detail & Related papers  (2023-11-06T14:34:36Z)
• Learning Locally Editable Virtual Humans [37.95173373011365]
  We propose a novel hybrid representation and end-to-end trainable network architecture to model fully editable neural avatars. At the core of our work lies a representation that combines the modeling power of neural fields with the ease of use and inherent 3D consistency of skinned meshes. Our method generates diverse detailed avatars and achieves better model fitting performance compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-04-28T23:06:17Z)
• AniPixel: Towards Animatable Pixel-Aligned Human Avatar [65.7175527782209]
  AniPixel is a novel animatable and generalizable human avatar reconstruction method. We propose a neural skinning field based on skeleton-driven deformation to establish the target-to-canonical and canonical-to-observation correspondences. Experiments show that AniPixel renders comparable novel views while delivering better novel pose animation results than state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-02-07T11:04:14Z)
• PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields [60.66412075837952]
  We present PaletteNeRF, a novel method for appearance editing of neural radiance fields (NeRF) based on 3D color decomposition. Our method decomposes the appearance of each 3D point into a linear combination of palette-based bases. We extend our framework with compressed semantic features for semantic-aware appearance editing.
  arXiv  Detail & Related papers  (2022-12-21T00:20:01Z)
• Vision Transformer for NeRF-Based View Synthesis from a Single Input Image [49.956005709863355]
  We propose to leverage both the global and local features to form an expressive 3D representation. To synthesize a novel view, we train a multilayer perceptron (MLP) network conditioned on the learned 3D representation to perform volume rendering. Our method can render novel views from only a single input image and generalize across multiple object categories using a single model.
  arXiv  Detail & Related papers  (2022-07-12T17:52:04Z)
• Neural Parameterization for Dynamic Human Head Editing [26.071370285285465]
  We present Neuralization (NeP), a hybrid representation that provides the advantages of both implicit and explicit methods. NeP is capable of photo-realistic rendering while allowing fine-grained editing of the scene geometry and appearance. The results show that the NeP achieves almost the same level of rendering accuracy while maintaining high editability.
  arXiv  Detail & Related papers  (2022-07-01T05:25:52Z)
• Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos [63.16888987770885]
  This paper addresses the challenge of reconstructing an animatable human model from a multi-view video. We introduce a pose-driven deformation field based on the linear blend skinning algorithm. We show that our approach significantly outperforms recent human modeling methods.
  arXiv  Detail & Related papers  (2022-03-15T17:56:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.