Continuous Piecewise-Affine Based Motion Model for Image Animation
- URL: http://arxiv.org/abs/2401.09146v1
- Date: Wed, 17 Jan 2024 11:40:05 GMT
- Title: Continuous Piecewise-Affine Based Motion Model for Image Animation
- Authors: Hexiang Wang, Fengqi Liu, Qianyu Zhou, Ran Yi, Xin Tan, Lizhuang Ma
- Abstract summary: Image animation aims to bring static images to life according to driving videos.
Recent unsupervised methods utilize affine and thin-plate spline transformations based on keypoints to transfer the motion in driving frames to the source image.
We propose to model motion from the source image to the driving frame in highly-expressive diffeo spaces.
- Score: 45.55812811136834
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Image animation aims to bring static images to life according to driving
videos and create engaging visual content that can be used for various purposes
such as animation, entertainment, and education. Recent unsupervised methods
utilize affine and thin-plate spline transformations based on keypoints to
transfer the motion in driving frames to the source image. However, limited by
the expressive power of the transformations used, these methods always produce
poor results when the gap between the motion in the driving frame and the
source image is large. To address this issue, we propose to model motion from
the source image to the driving frame in highly-expressive diffeomorphism
spaces. Firstly, we introduce Continuous Piecewise-Affine based (CPAB)
transformation to model the motion and present a well-designed inference
algorithm to generate CPAB transformation from control keypoints. Secondly, we
propose a SAM-guided keypoint semantic loss to further constrain the keypoint
extraction process and improve the semantic consistency between the
corresponding keypoints on the source and driving images. Finally, we design a
structure alignment loss to align the structure-related features extracted from
driving and generated images, thus helping the generator generate results that
are more consistent with the driving action. Extensive experiments on four
datasets demonstrate the effectiveness of our method against state-of-the-art
competitors quantitatively and qualitatively. Code will be publicly available
at: https://github.com/DevilPG/AAAI2024-CPABMM.
Related papers
- Distance Weighted Trans Network for Image Completion [52.318730994423106]
We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components.
CNNs are used to augment the local texture information of coarse priors.
DWT blocks are used to recover certain coarse textures and coherent visual structures.
arXiv Detail & Related papers (2023-10-11T12:46:11Z) - Pixel-Inconsistency Modeling for Image Manipulation Localization [63.54342601757723]
Digital image forensics plays a crucial role in image authentication and manipulation localization.
This paper presents a generalized and robust manipulation localization model through the analysis of pixel inconsistency artifacts.
Experiments show that our method successfully extracts inherent pixel-inconsistency forgery fingerprints.
arXiv Detail & Related papers (2023-09-30T02:54:51Z) - Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold [79.94300820221996]
DragGAN is a new way of controlling generative adversarial networks (GANs)
DragGAN allows anyone to deform an image with precise control over where pixels go, thus manipulating the pose, shape, expression, and layout of diverse categories such as animals, cars, humans, landscapes, etc.
Both qualitative and quantitative comparisons demonstrate the advantage of DragGAN over prior approaches in the tasks of image manipulation and point tracking.
arXiv Detail & Related papers (2023-05-18T13:41:25Z) - Motion and Appearance Adaptation for Cross-Domain Motion Transfer [36.98500700394921]
Motion transfer aims to transfer the motion of a driving video to a source image.
Traditional single domain motion transfer approaches often produce notable artifacts.
We propose a Motion and Appearance Adaptation (MAA) approach for cross-domain motion transfer.
arXiv Detail & Related papers (2022-09-29T03:24:47Z) - Motion Transformer for Unsupervised Image Animation [37.35527776043379]
Image animation aims to animate a source image by using motion learned from a driving video.
Current state-of-the-art methods typically use convolutional neural networks (CNNs) to predict motion information.
We propose a new method, the motion transformer, which is the first attempt to build a motion estimator based on a vision transformer.
arXiv Detail & Related papers (2022-09-28T12:04:58Z) - A Constrained Deformable Convolutional Network for Efficient Single
Image Dynamic Scene Blind Deblurring with Spatially-Variant Motion Blur
Kernels Estimation [12.744989551644744]
We propose a novel constrained deformable convolutional network (CDCN) for efficient single image dynamic scene blind deblurring.
CDCN simultaneously achieves accurate spatially-variant motion blur kernels estimation and the high-quality image restoration.
arXiv Detail & Related papers (2022-08-23T03:28:21Z) - DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation [56.514462874501675]
We propose a dynamic sparse attention based Transformer model to achieve fine-level matching with favorable efficiency.
The heart of our approach is a novel dynamic-attention unit, dedicated to covering the variation on the optimal number of tokens one position should focus on.
Experiments on three applications, pose-guided person image generation, edge-based face synthesis, and undistorted image style transfer, demonstrate that DynaST achieves superior performance in local details.
arXiv Detail & Related papers (2022-07-13T11:12:03Z) - Thin-Plate Spline Motion Model for Image Animation [9.591298403129532]
Image animation brings life to the static object in the source image according to the driving video.
Recent works attempt to perform motion transfer on arbitrary objects through unsupervised methods without using a priori knowledge.
It remains a significant challenge for current unsupervised methods when there is a large pose gap between the objects in the source and driving images.
arXiv Detail & Related papers (2022-03-27T18:40:55Z) - Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
Motion blur in an image may have practical interests in fundamental computer vision problems.
We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
arXiv Detail & Related papers (2021-03-04T12:45:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.