Interactive Rendering of Relightable and Animatable Gaussian Avatars

• URL: http://arxiv.org/abs/2407.10707v1
• Date: Mon, 15 Jul 2024 13:25:07 GMT
• Title: Interactive Rendering of Relightable and Animatable Gaussian Avatars
• Authors: Youyi Zhan, Tianjia Shao, He Wang, Yin Yang, Kun Zhou,
• Abstract summary: We propose a simple and efficient method to decouple body materials and lighting from multi-view or monocular avatar videos. Our method can render higher quality results at a faster speed on both synthetic and real datasets.
• Score: 37.73483372890271
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Creating relightable and animatable avatars from multi-view or monocular videos is a challenging task for digital human creation and virtual reality applications. Previous methods rely on neural radiance fields or ray tracing, resulting in slow training and rendering processes. By utilizing Gaussian Splatting, we propose a simple and efficient method to decouple body materials and lighting from sparse-view or monocular avatar videos, so that the avatar can be rendered simultaneously under novel viewpoints, poses, and lightings at interactive frame rates (6.9 fps). Specifically, we first obtain the canonical body mesh using a signed distance function and assign attributes to each mesh vertex. The Gaussians in the canonical space then interpolate from nearby body mesh vertices to obtain the attributes. We subsequently deform the Gaussians to the posed space using forward skinning, and combine the learnable environment light with the Gaussian attributes for shading computation. To achieve fast shadow modeling, we rasterize the posed body mesh from dense viewpoints to obtain the visibility. Our approach is not only simple but also fast enough to allow interactive rendering of avatar animation under environmental light changes. Experiments demonstrate that, compared to previous works, our method can render higher quality results at a faster speed on both synthetic and real datasets.

Related papers

• SplattingAvatar: Realistic Real-Time Human Avatars with Mesh-Embedded Gaussian Splatting [26.849406891462557]
  We present SplattingAvatar, a hybrid 3D representation of human avatars with Gaussian Splatting embedded on a triangle mesh. SplattingAvatar renders over 300 FPS on a modern GPU and 30 FPS on a mobile device.
  arXiv  Detail & Related papers  (2024-03-08T06:28:09Z)
• ASH: Animatable Gaussian Splats for Efficient and Photoreal Human Rendering [62.81677824868519]
  We propose an animatable Gaussian splatting approach for photorealistic rendering of dynamic humans in real-time. We parameterize the clothed human as animatable 3D Gaussians, which can be efficiently splatted into image space to generate the final rendering. We benchmark ASH with competing methods on pose-controllable avatars, demonstrating that our method outperforms existing real-time methods by a large margin and shows comparable or even better results than offline methods.
  arXiv  Detail & Related papers  (2023-12-10T17:07:37Z)
• Relightable Gaussian Codec Avatars [26.255161061306428]
  We present Relightable Gaussian Codec Avatars, a method to build high-fidelity relightable head avatars that can be animated to generate novel expressions. Our geometry model based on 3D Gaussians can capture 3D-consistent sub-millimeter details such as hair strands and pores on dynamic face sequences. We improve the fidelity of eye reflections and enable explicit gaze control by introducing relightable explicit eye models.
  arXiv  Detail & Related papers  (2023-12-06T18:59:58Z)
• FLARE: Fast Learning of Animatable and Relightable Mesh Avatars [64.48254296523977]
  Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. We introduce FLARE, a technique that enables the creation of animatable and relightable avatars from a single monocular video.
  arXiv  Detail & Related papers  (2023-10-26T16:13:00Z)
• Towards Practical Capture of High-Fidelity Relightable Avatars [60.25823986199208]
  TRAvatar is trained with dynamic image sequences captured in a Light Stage under varying lighting conditions. It can predict the appearance in real-time with a single forward pass, achieving high-quality relighting effects. Our framework achieves superior performance for photorealistic avatar animation and relighting.
  arXiv  Detail & Related papers  (2023-09-08T10:26:29Z)
• Efficient Meshy Neural Fields for Animatable Human Avatars [87.68529918184494]
  Efficiently digitizing high-fidelity animatable human avatars from videos is a challenging and active research topic. Recent rendering-based neural representations open a new way for human digitization with their friendly usability and photo-varying reconstruction quality. We present EMA, a method that Efficiently learns Meshy neural fields to reconstruct animatable human Avatars.
  arXiv  Detail & Related papers  (2023-03-23T00:15:34Z)
• PointAvatar: Deformable Point-based Head Avatars from Videos [103.43941945044294]
  PointAvatar is a deformable point-based representation that disentangles the source color into intrinsic albedo and normal-dependent shading. We show that our method is able to generate animatable 3D avatars using monocular videos from multiple sources.
  arXiv  Detail & Related papers  (2022-12-16T10:05:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.