RANA: Relightable Articulated Neural Avatars

• URL: http://arxiv.org/abs/2212.03237v1
• Date: Tue, 6 Dec 2022 18:59:31 GMT
• Title: RANA: Relightable Articulated Neural Avatars
• Authors: Umar Iqbal, Akin Caliskan, Koki Nagano, Sameh Khamis, Pavlo Molchanov, Jan Kautz
• Abstract summary: We propose RANA, a relightable and articulated neural avatar for the photorealistic synthesis of humans. We present a novel framework to model humans while disentangling their geometry, texture, and also lighting environment from monocular RGB videos.
• Score: 83.60081895984634
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose RANA, a relightable and articulated neural avatar for the photorealistic synthesis of humans under arbitrary viewpoints, body poses, and lighting. We only require a short video clip of the person to create the avatar and assume no knowledge about the lighting environment. We present a novel framework to model humans while disentangling their geometry, texture, and also lighting environment from monocular RGB videos. To simplify this otherwise ill-posed task we first estimate the coarse geometry and texture of the person via SMPL+D model fitting and then learn an articulated neural representation for photorealistic image generation. RANA first generates the normal and albedo maps of the person in any given target body pose and then uses spherical harmonics lighting to generate the shaded image in the target lighting environment. We also propose to pretrain RANA using synthetic images and demonstrate that it leads to better disentanglement between geometry and texture while also improving robustness to novel body poses. Finally, we also present a new photorealistic synthetic dataset, Relighting Humans, to quantitatively evaluate the performance of the proposed approach.

Related papers

• UV Gaussians: Joint Learning of Mesh Deformation and Gaussian Textures for Human Avatar Modeling [71.87807614875497]
  We propose UV Gaussians, which models the 3D human body by jointly learning mesh deformations and 2D UV-space Gaussian textures. We collect and process a new dataset of human motion, which includes multi-view images, scanned models, parametric model registration, and corresponding texture maps. Experimental results demonstrate that our method achieves state-of-the-art synthesis of novel view and novel pose.
  arXiv  Detail & Related papers  (2024-03-18T09:03:56Z)
• NECA: Neural Customizable Human Avatar [36.69012172745299]
  We introduce NECA, an approach capable of learning versatile human representation from monocular or sparse-view videos. The core of our approach is to represent humans in complementary dual spaces and predict disentangled neural fields of geometry, albedo, shadow, as well as an external lighting.
  arXiv  Detail & Related papers  (2024-03-15T14:23:06Z)
• Relightable and Animatable Neural Avatars from Videos [14.091229306680697]
  We propose a method to create relightable and animatable neural avatars. The key challenge is to disentangle the geometry, material of the clothed body, and lighting. Experiments on synthetic and real datasets show that our approach reconstructs high-quality geometry.
  arXiv  Detail & Related papers  (2023-12-20T09:39:55Z)
• SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance [37.85026590250023]
  We present SEEAvatar, a method for generating photorealistic 3D avatars from text. For geometry, we propose to constrain the optimized avatar in a decent global shape with a template avatar. For appearance generation, we use diffusion model enhanced by prompt engineering to guide a physically based rendering pipeline.
  arXiv  Detail & Related papers  (2023-12-13T14:48:35Z)
• Novel View Synthesis of Humans using Differentiable Rendering [50.57718384229912]
  We present a new approach for synthesizing novel views of people in new poses. Our synthesis makes use of diffuse Gaussian primitives that represent the underlying skeletal structure of a human. Rendering these primitives gives results in a high-dimensional latent image, which is then transformed into an RGB image by a decoder network.
  arXiv  Detail & Related papers  (2023-03-28T10:48:33Z)
• DINAR: Diffusion Inpainting of Neural Textures for One-Shot Human Avatars [7.777410338143783]
  We present an approach for creating realistic rigged fullbody avatars from single RGB images. Our method uses neural textures combined with the SMPL-X body model to achieve photo-realistic quality of avatars. In the experiments, our approach achieves state-of-the-art rendering quality and good generalization to new poses and viewpoints.
  arXiv  Detail & Related papers  (2023-03-16T15:04:10Z)
• Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation [76.96499178502759]
  Relighting aims to re-illuminate the person in the image as if the person appeared in an environment with the target lighting. Recent methods rely on deep learning to achieve high-quality results. We propose a new approach that can perform on par with the state-of-the-art (SOTA) relighting methods without requiring a light stage.
  arXiv  Detail & Related papers  (2022-09-21T17:15:58Z)
• Geometry-aware Single-image Full-body Human Relighting [37.381122678376805]
  Single-image human relighting aims to relight a target human under new lighting conditions by decomposing the input image into albedo, shape and lighting. Previous methods suffer from both the entanglement between albedo and lighting and the lack of hard shadows. Our framework is able to generate photo-realistic high-frequency shadows such as cast shadows under challenging lighting conditions.
  arXiv  Detail & Related papers  (2022-07-11T10:21:02Z)
• EgoRenderer: Rendering Human Avatars from Egocentric Camera Images [87.96474006263692]
  We present EgoRenderer, a system for rendering full-body neural avatars of a person captured by a wearable, egocentric fisheye camera. Rendering full-body avatars from such egocentric images come with unique challenges due to the top-down view and large distortions. We tackle these challenges by decomposing the rendering process into several steps, including texture synthesis, pose construction, and neural image translation.
  arXiv  Detail & Related papers  (2021-11-24T18:33:02Z)
• Neural Re-Rendering of Humans from a Single Image [80.53438609047896]
  We propose a new method for neural re-rendering of a human under a novel user-defined pose and viewpoint. Our algorithm represents body pose and shape as a parametric mesh which can be reconstructed from a single image.
  arXiv  Detail & Related papers  (2021-01-11T18:53:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.