Relightable and Animatable Neural Avatars from Videos

• URL: http://arxiv.org/abs/2312.12877v1
• Date: Wed, 20 Dec 2023 09:39:55 GMT
• Title: Relightable and Animatable Neural Avatars from Videos
• Authors: Wenbin Lin, Chengwei Zheng, Jun-Hai Yong, Feng Xu
• Abstract summary: We propose a method to create relightable and animatable neural avatars. The key challenge is to disentangle the geometry, material of the clothed body, and lighting. Experiments on synthetic and real datasets show that our approach reconstructs high-quality geometry.
• Score: 14.091229306680697
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Lightweight creation of 3D digital avatars is a highly desirable but challenging task. With only sparse videos of a person under unknown illumination, we propose a method to create relightable and animatable neural avatars, which can be used to synthesize photorealistic images of humans under novel viewpoints, body poses, and lighting. The key challenge here is to disentangle the geometry, material of the clothed body, and lighting, which becomes more difficult due to the complex geometry and shadow changes caused by body motions. To solve this ill-posed problem, we propose novel techniques to better model the geometry and shadow changes. For geometry change modeling, we propose an invertible deformation field, which helps to solve the inverse skinning problem and leads to better geometry quality. To model the spatial and temporal varying shading cues, we propose a pose-aware part-wise light visibility network to estimate light occlusion. Extensive experiments on synthetic and real datasets show that our approach reconstructs high-quality geometry and generates realistic shadows under different body poses. Code and data are available at \url{https://wenbin-lin.github.io/RelightableAvatar-page/}.

Related papers

• MeshAvatar: Learning High-quality Triangular Human Avatars from Multi-view Videos [41.45299653187577]
  We present a novel pipeline for learning high-quality triangular human avatars from multi-view videos. Our method represents the avatar with an explicit triangular mesh extracted from an implicit SDF field. We incorporate physics-based rendering to accurately decompose geometry and texture.
  arXiv  Detail & Related papers  (2024-07-11T11:37:51Z)
• Relightable Neural Actor with Intrinsic Decomposition and Pose Control [80.06094206522668]
  We propose Relightable Neural Actor, a new video-based method for learning a pose-driven neural human model that can be relighted. For training, our method solely requires a multi-view recording of the human under a known, but static lighting condition. To evaluate our approach in real-world scenarios, we collect a new dataset with four identities recorded under different light conditions, indoors and outdoors.
  arXiv  Detail & Related papers  (2023-12-18T14:30:13Z)
• SEEAvatar: Photorealistic Text-to-3D Avatar Generation with Constrained Geometry and Appearance [37.85026590250023]
  We present SEEAvatar, a method for generating photorealistic 3D avatars from text. For geometry, we propose to constrain the optimized avatar in a decent global shape with a template avatar. For appearance generation, we use diffusion model enhanced by prompt engineering to guide a physically based rendering pipeline.
  arXiv  Detail & Related papers  (2023-12-13T14:48:35Z)
• TriHuman : A Real-time and Controllable Tri-plane Representation for Detailed Human Geometry and Appearance Synthesis [76.73338151115253]
  TriHuman is a novel human-tailored, deformable, and efficient tri-plane representation. We non-rigidly warp global ray samples into our undeformed tri-plane texture space. We show how such a tri-plane feature representation can be conditioned on the skeletal motion to account for dynamic appearance and geometry changes.
  arXiv  Detail & Related papers  (2023-12-08T16:40:38Z)
• FLARE: Fast Learning of Animatable and Relightable Mesh Avatars [64.48254296523977]
  Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. We introduce FLARE, a technique that enables the creation of animatable and relightable avatars from a single monocular video.
  arXiv  Detail & Related papers  (2023-10-26T16:13:00Z)
• Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization [91.52882218901627]
  We propose a novel method for constructing implicit 3D morphable face models that are both generalizable and intuitive for editing. Our method improves upon photo-realism, geometry, and expression accuracy compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-04T17:58:40Z)
• RANA: Relightable Articulated Neural Avatars [83.60081895984634]
  We propose RANA, a relightable and articulated neural avatar for the photorealistic synthesis of humans. We present a novel framework to model humans while disentangling their geometry, texture, and also lighting environment from monocular RGB videos.
  arXiv  Detail & Related papers  (2022-12-06T18:59:31Z)
• SPIDR: SDF-based Neural Point Fields for Illumination and Deformation [4.246563675883777]
  We introduce SPIDR, a new hybrid neural SDF representation. We propose a novel neural implicit model to learn environment light. We demonstrate the effectiveness of SPIDR in enabling high quality geometry editing with more accurate updates to the illumination of the scene.
  arXiv  Detail & Related papers  (2022-10-15T23:34:53Z)
• Geometry-aware Single-image Full-body Human Relighting [37.381122678376805]
  Single-image human relighting aims to relight a target human under new lighting conditions by decomposing the input image into albedo, shape and lighting. Previous methods suffer from both the entanglement between albedo and lighting and the lack of hard shadows. Our framework is able to generate photo-realistic high-frequency shadows such as cast shadows under challenging lighting conditions.
  arXiv  Detail & Related papers  (2022-07-11T10:21:02Z)
• Generalizable Neural Performer: Learning Robust Radiance Fields for Human Novel View Synthesis [52.720314035084215]
  This work targets at using a general deep learning framework to synthesize free-viewpoint images of arbitrary human performers. We present a simple yet powerful framework, named Generalizable Neural Performer (GNR), that learns a generalizable and robust neural body representation. Experiments on GeneBody-1.0 and ZJU-Mocap show better robustness of our methods than recent state-of-the-art generalizable methods.
  arXiv  Detail & Related papers  (2022-04-25T17:14:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.