Single Source One Shot Reenactment using Weighted motion From Paired Feature Points

• URL: http://arxiv.org/abs/2104.03117v1
• Date: Wed, 7 Apr 2021 13:45:34 GMT
• Title: Single Source One Shot Reenactment using Weighted motion From Paired Feature Points
• Authors: Soumya Tripathy, Juho Kannala, Esa Rahtu
• Abstract summary: We propose a new (face) reenactment model that learns shape-independent motion features in a self-supervised setup. The model faithfully transfers the driving motion to the source while retaining the source identity intact.
• Score: 26.210285908770377
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Image reenactment is a task where the target object in the source image imitates the motion represented in the driving image. One of the most common reenactment tasks is face image animation. The major challenge in the current face reenactment approaches is to distinguish between facial motion and identity. For this reason, the previous models struggle to produce high-quality animations if the driving and source identities are different (cross-person reenactment). We propose a new (face) reenactment model that learns shape-independent motion features in a self-supervised setup. The motion is represented using a set of paired feature points extracted from the source and driving images simultaneously. The model is generalised to multiple reenactment tasks including faces and non-face objects using only a single source image. The extensive experiments show that the model faithfully transfers the driving motion to the source while retaining the source identity intact.

Related papers

• FSRT: Facial Scene Representation Transformer for Face Reenactment from Factorized Appearance, Head-pose, and Facial Expression Features [17.531847357428454]
  The task of face reenactment is to transfer the head motion and facial expressions from a driving video to the appearance of a source image. Most existing methods are CNN-based and estimate optical flow from the source image to the current driving frame. We propose a transformer-based encoder for computing a set-latent representation of the source image.
  arXiv  Detail & Related papers  (2024-04-15T12:37:26Z)
• HyperReenact: One-Shot Reenactment via Jointly Learning to Refine and Retarget Faces [47.27033282706179]
  We present our method for neural face reenactment, called HyperReenact, that aims to generate realistic talking head images of a source identity. Our method operates under the one-shot setting (i.e., using a single source frame) and allows for cross-subject reenactment, without requiring subject-specific fine-tuning. We compare our method both quantitatively and qualitatively against several state-of-the-art techniques on the standard benchmarks of VoxCeleb1 and VoxCeleb2.
  arXiv  Detail & Related papers  (2023-07-20T11:59:42Z)
• StyleMask: Disentangling the Style Space of StyleGAN2 for Neural Face Reenactment [47.27033282706179]
  We propose a framework that learns to disentangle the identity characteristics of the face from its pose. We show that the proposed method produces higher quality results even on extreme pose variations.
  arXiv  Detail & Related papers  (2022-09-27T13:22:35Z)
• Everything's Talkin': Pareidolia Face Reenactment [119.49707201178633]
  Pareidolia Face Reenactment is defined as animating a static illusory face to move in tandem with a human face in the video. For the large differences between pareidolia face reenactment and traditional human face reenactment, shape variance and texture variance are introduced. We propose a novel Parametric Unsupervised Reenactment Algorithm to tackle these two challenges.
  arXiv  Detail & Related papers  (2021-04-07T11:19:13Z)
• LI-Net: Large-Pose Identity-Preserving Face Reenactment Network [14.472453602392182]
  We propose a large-pose identity-preserving face reenactment network, LI-Net. Specifically, the Landmark Transformer is adopted to adjust driving landmark images. The Face Rotation Module and the Expression Enhancing Generator decouple the transformed landmark image into pose and expression features, and reenact those attributes separately to generate identity-preserving faces.
  arXiv  Detail & Related papers  (2021-04-07T01:41:21Z)
• Head2HeadFS: Video-based Head Reenactment with Few-shot Learning [64.46913473391274]
  Head reenactment is a challenging task, which aims at transferring the entire head pose from a source person to a target. We propose head2headFS, a novel easily adaptable pipeline for head reenactment. Our video-based rendering network is fine-tuned under a few-shot learning strategy, using only a few samples.
  arXiv  Detail & Related papers  (2021-03-30T10:19:41Z)
• Image Animation with Perturbed Masks [95.94432031144716]
  We present a novel approach for image-animation of a source image by a driving video, both depicting the same type of object. We do not assume the existence of pose models and our method is able to animate arbitrary objects without the knowledge of the object's structure.
  arXiv  Detail & Related papers  (2020-11-13T14:17:17Z)
• FACEGAN: Facial Attribute Controllable rEenactment GAN [24.547319786399743]
  Face reenactment is a popular animation method where the person's identity is taken from the source image and the facial motion from the driving image. Recent works have demonstrated high quality results by combining the facial landmark based motion representations with the generative adversarial networks. We propose a novel Facial Attribute Controllable rEenactment GAN (FACEGAN), which transfers the facial motion from the driving face via the Action Unit (AU) representation.
  arXiv  Detail & Related papers  (2020-11-09T14:04:15Z)
• FaR-GAN for One-Shot Face Reenactment [20.894596219099164]
  We present a one-shot face reenactment model, FaR-GAN, that takes only one face image of any given source identity and a target expression as input. The proposed method makes no assumptions about the source identity, facial expression, head pose, or even image background.
  arXiv  Detail & Related papers  (2020-05-13T16:15:37Z)
• First Order Motion Model for Image Animation [90.712718329677]
  Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate.
  arXiv  Detail & Related papers  (2020-02-29T07:08:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.