APB2FaceV2: Real-Time Audio-Guided Multi-Face Reenactment

• URL: http://arxiv.org/abs/2010.13017v1
• Date: Sun, 25 Oct 2020 02:30:09 GMT
• Title: APB2FaceV2: Real-Time Audio-Guided Multi-Face Reenactment
• Abstract summary: We propose a novel emphReal-time emphAudio-guided emphMulti-face reenactment approach named emphAPB2FaceV2. It can reenact different target faces among multiple persons with corresponding reference face and drive audio signal as inputs.
• Score: 25.619906272417303
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-guided face reenactment aims to generate a photorealistic face that has matched facial expression with the input audio. However, current methods can only reenact a special person once the model is trained or need extra operations such as 3D rendering and image post-fusion on the premise of generating vivid faces. To solve the above challenge, we propose a novel \emph{R}eal-time \emph{A}udio-guided \emph{M}ulti-face reenactment approach named \emph{APB2FaceV2}, which can reenact different target faces among multiple persons with corresponding reference face and drive audio signal as inputs. Enabling the model to be trained end-to-end and have a faster speed, we design a novel module named Adaptive Convolution (AdaConv) to infuse audio information into the network, as well as adopt a lightweight network as our backbone so that the network can run in real time on CPU and GPU. Comparison experiments prove the superiority of our approach than existing state-of-the-art methods, and further experiments demonstrate that our method is efficient and flexible for practical applications https://github.com/zhangzjn/APB2FaceV2

Related papers

• Mask-Free Audio-driven Talking Face Generation for Enhanced Visual Quality and Identity Preservation [54.52905471078152]
  We propose a mask-free talking face generation approach while maintaining the 2D-based face editing task.<n>We transform the input images to have closed mouths, using a two-step landmark-based approach trained in an unpaired manner.
  arXiv  Detail & Related papers  (2025-07-28T16:03:36Z)
• OSDFace: One-Step Diffusion Model for Face Restoration [72.5045389847792]
  Diffusion models have demonstrated impressive performance in face restoration. We propose OSDFace, a novel one-step diffusion model for face restoration. Results demonstrate that OSDFace surpasses current state-of-the-art (SOTA) methods in both visual quality and quantitative metrics.
  arXiv  Detail & Related papers  (2024-11-26T07:07:48Z)
• GSmoothFace: Generalized Smooth Talking Face Generation via Fine Grained 3D Face Guidance [83.43852715997596]
  GSmoothFace is a novel two-stage generalized talking face generation model guided by a fine-grained 3d face model. It can synthesize smooth lip dynamics while preserving the speaker's identity. Both quantitative and qualitative experiments confirm the superiority of our method in terms of realism, lip synchronization, and visual quality.
  arXiv  Detail & Related papers  (2023-12-12T16:00:55Z)
• Parametric Implicit Face Representation for Audio-Driven Facial Reenactment [52.33618333954383]
  We propose a novel audio-driven facial reenactment framework that is both controllable and can generate high-quality talking heads. Specifically, our parametric implicit representation parameterizes the implicit representation with interpretable parameters of 3D face models. Our method can generate more realistic results than previous methods with greater fidelity to the identities and talking styles of speakers.
  arXiv  Detail & Related papers  (2023-06-13T07:08:22Z)
• GeneFace++: Generalized and Stable Real-Time Audio-Driven 3D Talking Face Generation [71.73912454164834]
  A modern talking face generation method is expected to achieve the goals of generalized audio-lip synchronization, good video quality, and high system efficiency. NeRF has become a popular technique in this field since it could achieve high-fidelity and 3D-consistent talking face generation with a few-minute-long training video. We propose GeneFace++ to handle these challenges by utilizing the rendering pitch contour as an auxiliary feature and introducing a temporal loss in the facial motion prediction process.
  arXiv  Detail & Related papers  (2023-05-01T12:24:09Z)
• Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis [90.43371339871105]
  We propose Dynamic Facial Radiance Fields (DFRF) for few-shot talking head synthesis. DFRF conditions face radiance field on 2D appearance images to learn the face prior. Experiments show DFRF can synthesize natural and high-quality audio-driven talking head videos for novel identities with only 40k iterations.
  arXiv  Detail & Related papers  (2022-07-24T16:46:03Z)
• Video2StyleGAN: Encoding Video in Latent Space for Manipulation [63.03250800510085]
  We propose a novel network to encode face videos into the latent space of StyleGAN for semantic face video manipulation. Our approach can significantly outperform existing single image methods, while achieving real-time (66 fps) speed.
  arXiv  Detail & Related papers  (2022-06-27T06:48:15Z)
• Synthetic Expressions are Better Than Real for Learning to Detect Facial Actions [4.4532095214807965]
  Our approach reconstructs the 3D shape of the face from each video frame, aligns the 3D mesh to a canonical view, and then trains a GAN-based network to synthesize novel images with facial action units of interest. The network trained on synthesized facial expressions outperformed the one trained on actual facial expressions and surpassed current state-of-the-art approaches.
  arXiv  Detail & Related papers  (2020-10-21T13:11:45Z)
• APB2Face: Audio-guided face reenactment with auxiliary pose and blink signals [15.89326520522585]
  Audio-guided face reenactment aims at generating photorealistic faces using audio information while maintaining the same facial movement as when speaking to a real person. Existing methods can not generate vivid face images or only reenact low-resolution faces, which limits the application value. We propose a novel deep neural network named APB2Face, which consists of GeometryPredictor and FaceReenactor modules.
  arXiv  Detail & Related papers  (2020-04-30T03:44:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.