Image Animation with Perturbed Masks

• URL: http://arxiv.org/abs/2011.06922v3
• Date: Tue, 29 Mar 2022 09:30:26 GMT
• Title: Image Animation with Perturbed Masks
• Authors: Yoav Shalev, Lior Wolf
• Abstract summary: We present a novel approach for image-animation of a source image by a driving video, both depicting the same type of object. We do not assume the existence of pose models and our method is able to animate arbitrary objects without the knowledge of the object's structure.
• Score: 95.94432031144716
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a novel approach for image-animation of a source image by a driving video, both depicting the same type of object. We do not assume the existence of pose models and our method is able to animate arbitrary objects without the knowledge of the object's structure. Furthermore, both, the driving video and the source image are only seen during test-time. Our method is based on a shared mask generator, which separates the foreground object from its background, and captures the object's general pose and shape. To control the source of the identity of the output frame, we employ perturbations to interrupt the unwanted identity information on the driver's mask. A mask-refinement module then replaces the identity of the driver with the identity of the source. Conditioned on the source image, the transformed mask is then decoded by a multi-scale generator that renders a realistic image, in which the content of the source frame is animated by the pose in the driving video. Due to the lack of fully supervised data, we train on the task of reconstructing frames from the same video the source image is taken from. Our method is shown to greatly outperform the state-of-the-art methods on multiple benchmarks. Our code and samples are available at https://github.com/itsyoavshalev/Image-Animation-with-Perturbed-Masks.

Related papers

• MegActor: Harness the Power of Raw Video for Vivid Portrait Animation [16.013989935948114]
  MegActor harnesses the power of the raw videos for vivid portrait animation. We introduce a synthetic data generation framework for creating videos with consistent motion and expressions but inconsistent IDs. We further style transfer the appearance of the reference image to the driving video to eliminate the influence of facial details in the driving videos.
  arXiv  Detail & Related papers  (2024-05-31T14:33:13Z)
• Lazy Diffusion Transformer for Interactive Image Editing [79.75128130739598]
  We introduce a novel diffusion transformer, LazyDiffusion, that generates partial image updates efficiently. Our approach targets interactive image editing applications in which, starting from a blank canvas or an image, a user specifies a sequence of localized image modifications.
  arXiv  Detail & Related papers  (2024-04-18T17:59:27Z)
• Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos [32.74215702447293]
  We propose a generative model that synthesizes a photorealistic output that follows a prescribed layout. Our method transfers fine details from the original image and preserves the identity of its parts. We show that by using simple segmentations and coarse 2D manipulations, we can synthesize a photorealistic edit faithful to the user's input.
  arXiv  Detail & Related papers  (2024-03-19T17:59:58Z)
• Siamese Masked Autoencoders [76.35448665609998]
  We present Siamese Masked Autoencoders (SiamMAE) for learning visual correspondence from videos. SiamMAE operates on pairs of randomly sampled video frames and asymmetrically masks them. It outperforms state-of-the-art self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation tasks.
  arXiv  Detail & Related papers  (2023-05-23T17:59:46Z)
• Thin-Plate Spline Motion Model for Image Animation [9.591298403129532]
  Image animation brings life to the static object in the source image according to the driving video. Recent works attempt to perform motion transfer on arbitrary objects through unsupervised methods without using a priori knowledge. It remains a significant challenge for current unsupervised methods when there is a large pose gap between the objects in the source and driving images.
  arXiv  Detail & Related papers  (2022-03-27T18:40:55Z)
• Image Animation with Keypoint Mask [0.0]
  Motion transfer is the task of synthesizing future video frames of a single source image according to the motion from a given driving video. In this work, we extract the structure from a keypoint heatmap, without an explicit motion representation. Then, the structures from the image and the video are extracted to warp the image according to the video, by a deep generator.
  arXiv  Detail & Related papers  (2021-12-20T11:35:06Z)
• Self-Supervised Equivariant Scene Synthesis from Video [84.15595573718925]
  We propose a framework to learn scene representations from video that are automatically delineated into background, characters, and animations. After training, we can manipulate image encodings in real time to create unseen combinations of the delineated components. We demonstrate results on three datasets: Moving MNIST with backgrounds, 2D video game sprites, and Fashion Modeling.
  arXiv  Detail & Related papers  (2021-02-01T14:17:31Z)
• Learned Equivariant Rendering without Transformation Supervision [105.15592625987911]
  We propose a framework to learn scene representations from video that are automatically delineated into objects and background. After training, we can manipulate and render the scenes in real time to create unseen combinations of objects, transformations, and backgrounds.
  arXiv  Detail & Related papers  (2020-11-11T14:05:05Z)
• First Order Motion Model for Image Animation [90.712718329677]
  Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video. Our framework addresses this problem without using any annotation or prior information about the specific object to animate.
  arXiv  Detail & Related papers  (2020-02-29T07:08:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.