FNeVR: Neural Volume Rendering for Face Animation

• URL: http://arxiv.org/abs/2209.10340v1
• Date: Wed, 21 Sep 2022 13:18:59 GMT
• Title: FNeVR: Neural Volume Rendering for Face Animation
• Authors: Bohan Zeng, Boyu Liu, Hong Li, Xuhui Liu, Jianzhuang Liu, Dapeng Chen, Wei Peng, Baochang Zhang
• Abstract summary: We propose a Face Neural Volume Rendering (FNeVR) network to explore the potential of 2D motion warping and 3D volume rendering. In FNeVR, we design a 3D Face Volume Rendering (FVR) module to enhance the facial details for image rendering. We also design a lightweight pose editor, enabling FNeVR to edit the facial pose in a simple yet effective way.
• Score: 53.92664037596834
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Face animation, one of the hottest topics in computer vision, has achieved a promising performance with the help of generative models. However, it remains a critical challenge to generate identity preserving and photo-realistic images due to the sophisticated motion deformation and complex facial detail modeling. To address these problems, we propose a Face Neural Volume Rendering (FNeVR) network to fully explore the potential of 2D motion warping and 3D volume rendering in a unified framework. In FNeVR, we design a 3D Face Volume Rendering (FVR) module to enhance the facial details for image rendering. Specifically, we first extract 3D information with a well-designed architecture, and then introduce an orthogonal adaptive ray-sampling module for efficient rendering. We also design a lightweight pose editor, enabling FNeVR to edit the facial pose in a simple yet effective way. Extensive experiments show that our FNeVR obtains the best overall quality and performance on widely used talking-head benchmarks.

Related papers

• G3FA: Geometry-guided GAN for Face Animation [14.488117084637631]
  We introduce Geometry-guided GAN for Face Animation (G3FA) to tackle this limitation. Our novel approach empowers the face animation model to incorporate 3D information using only 2D images. In our face reenactment model, we leverage 2D motion warping to capture motion dynamics.
  arXiv  Detail & Related papers  (2024-08-23T13:13:24Z)
• NeRFFaceSpeech: One-shot Audio-driven 3D Talking Head Synthesis via Generative Prior [5.819784482811377]
  We propose a novel method, NeRFFaceSpeech, which enables to produce high-quality 3D-aware talking head. Our method can craft a 3D-consistent facial feature space corresponding to a single image. We also introduce LipaintNet that can replenish the lacking information in the inner-mouth area.
  arXiv  Detail & Related papers  (2024-05-09T13:14:06Z)
• FaceFolds: Meshed Radiance Manifolds for Efficient Volumetric Rendering of Dynamic Faces [21.946327323788275]
  3D rendering of dynamic face is a challenging problem. We present a novel representation that enables high-quality rendering of an actor's dynamic facial performances.
  arXiv  Detail & Related papers  (2024-04-22T00:44:13Z)
• Versatile Face Animator: Driving Arbitrary 3D Facial Avatar in RGBD Space [38.940128217895115]
  We propose Versatile Face Animator, which combines facial motion capture with motion in an end-to-end manner, eliminating the need for blendshapes or rigs. Our method has the following two main characteristics: 1) we propose an RGBD animation module to learn facial motion from raw RGBD videos by hierarchical motion dictionaries and animate RGBD images rendered from 3D facial mesh coarse-to-fine, enabling facial animation on arbitrary 3D characters. Comprehensive experiments demonstrate the effectiveness of our proposed framework in generating impressive 3D facial animation results.
  arXiv  Detail & Related papers  (2023-08-11T11:29:01Z)
• Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives [70.32817882783608]
  We present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. Unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points.
  arXiv  Detail & Related papers  (2023-07-11T17:58:31Z)
• High-fidelity Facial Avatar Reconstruction from Monocular Video with Generative Priors [29.293166730794606]
  We propose a new method for NeRF-based facial avatar reconstruction that utilizes 3D-aware generative prior. Compared with existing works, we obtain superior novel view synthesis results and faithfully face reenactment performance.
  arXiv  Detail & Related papers  (2022-11-28T04:49:46Z)
• HVTR: Hybrid Volumetric-Textural Rendering for Human Avatars [65.82222842213577]
  We propose a novel neural rendering pipeline, which synthesizes virtual human avatars from arbitrary poses efficiently and at high quality. First, we learn to encode articulated human motions on a dense UV manifold of the human body surface. We then leverage the encoded information on the UV manifold to construct a 3D volumetric representation.
  arXiv  Detail & Related papers  (2021-12-19T17:34:15Z)
• PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering [56.762094966235566]
  A Portrait Image Neural Renderer is proposed to control the face motions with the parameters of three-dimensional morphable face models. The proposed model can generate photo-realistic portrait images with accurate movements according to intuitive modifications. Our model can generate coherent videos with convincing movements from only a single reference image and a driving audio stream.
  arXiv  Detail & Related papers  (2021-09-17T07:24:16Z)
• Inverting Generative Adversarial Renderer for Face Reconstruction [58.45125455811038]
  In this work, we introduce a novel Generative Adversa Renderer (GAR) GAR learns to model the complicated real-world image, instead of relying on the graphics rules, it is capable of producing realistic images. Our method achieves state-of-the-art performances on multiple face reconstruction.
  arXiv  Detail & Related papers  (2021-05-06T04:16:06Z)
• Pixel Codec Avatars [99.36561532588831]
  Pixel Codec Avatars (PiCA) is a deep generative model of 3D human faces. On a single Oculus Quest 2 mobile VR headset, 5 avatars are rendered in realtime in the same scene.
  arXiv  Detail & Related papers  (2021-04-09T23:17:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.