First Order Motion Model for Image Animation

• URL: http://arxiv.org/abs/2003.00196v3
• Date: Thu, 1 Oct 2020 15:26:15 GMT
• Title: First Order Motion Model for Image Animation
• Authors: Aliaksandr Siarohin, St\'ephane Lathuili\`ere, Sergey Tulyakov, Elisa Ricci and Nicu Sebe
• Abstract summary: Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate.
• Score: 90.712718329677
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate. Once trained on a set of videos depicting objects of the same category (e.g. faces, human bodies), our method can be applied to any object of this class. To achieve this, we decouple appearance and motion information using a self-supervised formulation. To support complex motions, we use a representation consisting of a set of learned keypoints along with their local affine transformations. A generator network models occlusions arising during target motions and combines the appearance extracted from the source image and the motion derived from the driving video. Our framework scores best on diverse benchmarks and on a variety of object categories. Our source code is publicly available.

Related papers

• Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics [67.97235923372035]
  We present Puppet-Master, an interactive video generative model that can serve as a motion prior for part-level dynamics. At test time, given a single image and a sparse set of motion trajectories, Puppet-Master can synthesize a video depicting realistic part-level motion faithful to the given drag interactions.
  arXiv  Detail & Related papers  (2024-08-08T17:59:38Z)
• Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversion [9.134743677331517]
  We propose a pre-trained image-to-video model to disentangle appearance from motion. Our method, called motion-textual inversion, leverages our observation that image-to-video models extract appearance mainly from the (latent) image input. By operating on an inflated motion-text embedding containing multiple text/image embedding tokens per frame, we achieve a high temporal motion granularity. Our approach does not require spatial alignment between the motion reference video and target image, generalizes across various domains, and can be applied to various tasks.
  arXiv  Detail & Related papers  (2024-08-01T10:55:20Z)
• AnimateZoo: Zero-shot Video Generation of Cross-Species Animation via Subject Alignment [64.02822911038848]
  We present AnimateZoo, a zero-shot diffusion-based video generator to produce animal animations. Key technique used in our AnimateZoo is subject alignment, which includes two steps. Our model is capable of generating videos characterized by accurate movements, consistent appearance, and high-fidelity frames.
  arXiv  Detail & Related papers  (2024-04-07T12:57:41Z)
• DragAPart: Learning a Part-Level Motion Prior for Articulated Objects [67.97235923372035]
  We introduce DragAPart, a method that generates a new image of the same object that responds to the action of the drags. Compared to prior motion-controlled generators, we demonstrate much better part-level motion understanding.
  arXiv  Detail & Related papers  (2024-03-22T17:58:59Z)
• MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model [74.84435399451573]
  This paper studies the human image animation task, which aims to generate a video of a certain reference identity following a particular motion sequence. Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion. We introduce MagicAnimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity.
  arXiv  Detail & Related papers  (2023-11-27T18:32:31Z)
• Image Animation with Keypoint Mask [0.0]
  Motion transfer is the task of synthesizing future video frames of a single source image according to the motion from a given driving video. In this work, we extract the structure from a keypoint heatmap, without an explicit motion representation. Then, the structures from the image and the video are extracted to warp the image according to the video, by a deep generator.
  arXiv  Detail & Related papers  (2021-12-20T11:35:06Z)
• Motion Representations for Articulated Animation [34.54825980226596]
  We propose novel motion representations for animating articulated objects consisting of distinct parts. In a completely unsupervised manner, our method identifies object parts, tracks them in a driving video, and infers their motions by considering their principal axes. Our model can animate a variety of objects, surpassing previous methods by a large margin on existing benchmarks.
  arXiv  Detail & Related papers  (2021-04-22T18:53:56Z)
• Image Animation with Perturbed Masks [95.94432031144716]
  We present a novel approach for image-animation of a source image by a driving video, both depicting the same type of object. We do not assume the existence of pose models and our method is able to animate arbitrary objects without the knowledge of the object's structure.
  arXiv  Detail & Related papers  (2020-11-13T14:17:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.