Thin-Plate Spline Motion Model for Image Animation
- URL: http://arxiv.org/abs/2203.14367v2
- Date: Tue, 29 Mar 2022 03:06:26 GMT
- Title: Thin-Plate Spline Motion Model for Image Animation
- Authors: Jian Zhao and Hui Zhang
- Abstract summary: Image animation brings life to the static object in the source image according to the driving video.
Recent works attempt to perform motion transfer on arbitrary objects through unsupervised methods without using a priori knowledge.
It remains a significant challenge for current unsupervised methods when there is a large pose gap between the objects in the source and driving images.
- Score: 9.591298403129532
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Image animation brings life to the static object in the source image
according to the driving video. Recent works attempt to perform motion transfer
on arbitrary objects through unsupervised methods without using a priori
knowledge. However, it remains a significant challenge for current unsupervised
methods when there is a large pose gap between the objects in the source and
driving images. In this paper, a new end-to-end unsupervised motion transfer
framework is proposed to overcome such issue. Firstly, we propose thin-plate
spline motion estimation to produce a more flexible optical flow, which warps
the feature maps of the source image to the feature domain of the driving
image. Secondly, in order to restore the missing regions more realistically, we
leverage multi-resolution occlusion masks to achieve more effective feature
fusion. Finally, additional auxiliary loss functions are designed to ensure
that there is a clear division of labor in the network modules, encouraging the
network to generate high-quality images. Our method can animate a variety of
objects, including talking faces, human bodies, and pixel animations.
Experiments demonstrate that our method performs better on most benchmarks than
the state of the art with visible improvements in pose-related metrics.
Related papers
- DiffUHaul: A Training-Free Method for Object Dragging in Images [78.93531472479202]
We propose a training-free method, dubbed DiffUHaul, for the object dragging task.
We first apply attention masking in each denoising step to make the generation more disentangled across different objects.
In the early denoising steps, we interpolate the attention features between source and target images to smoothly fuse new layouts with the original appearance.
arXiv Detail & Related papers (2024-06-03T17:59:53Z) - Continuous Piecewise-Affine Based Motion Model for Image Animation [45.55812811136834]
Image animation aims to bring static images to life according to driving videos.
Recent unsupervised methods utilize affine and thin-plate spline transformations based on keypoints to transfer the motion in driving frames to the source image.
We propose to model motion from the source image to the driving frame in highly-expressive diffeo spaces.
arXiv Detail & Related papers (2024-01-17T11:40:05Z) - Unsupervised Multi-object Segmentation by Predicting Probable Motion
Patterns [92.80981308407098]
We propose a new approach to learn to segment multiple image objects without manual supervision.
The method can extract objects form still images, but uses videos for supervision.
We show state-of-the-art unsupervised object segmentation performance on simulated and real-world benchmarks.
arXiv Detail & Related papers (2022-10-21T17:57:05Z) - Animation from Blur: Multi-modal Blur Decomposition with Motion Guidance [83.25826307000717]
We study the challenging problem of recovering detailed motion from a single motion-red image.
Existing solutions to this problem estimate a single image sequence without considering the motion ambiguity for each region.
In this paper, we explicitly account for such motion ambiguity, allowing us to generate multiple plausible solutions all in sharp detail.
arXiv Detail & Related papers (2022-07-20T18:05:53Z) - The Right Spin: Learning Object Motion from Rotation-Compensated Flow
Fields [61.664963331203666]
How humans perceive moving objects is a longstanding research question in computer vision.
One approach to the problem is to teach a deep network to model all of these effects.
We present a novel probabilistic model to estimate the camera's rotation given the motion field.
arXiv Detail & Related papers (2022-02-28T22:05:09Z) - Differential Motion Evolution for Fine-Grained Motion Deformation in
Unsupervised Image Animation [41.85199775016731]
We introduce DiME, an end-to-end unsupervised motion transfer framework.
By capturing the motion transfer with an ordinary differential equation (ODE), it helps to regularize the motion field.
We also propose a natural extension to the ODE idea, which is that DiME can easily leverage multiple different views of the source object whenever they are available.
arXiv Detail & Related papers (2021-10-09T22:44:30Z) - Motion Representations for Articulated Animation [34.54825980226596]
We propose novel motion representations for animating articulated objects consisting of distinct parts.
In a completely unsupervised manner, our method identifies object parts, tracks them in a driving video, and infers their motions by considering their principal axes.
Our model can animate a variety of objects, surpassing previous methods by a large margin on existing benchmarks.
arXiv Detail & Related papers (2021-04-22T18:53:56Z) - Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
Motion blur in an image may have practical interests in fundamental computer vision problems.
We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
arXiv Detail & Related papers (2021-03-04T12:45:18Z) - Cross-Identity Motion Transfer for Arbitrary Objects through
Pose-Attentive Video Reassembling [40.20163225821707]
Given a source image and a driving video, our networks animate the subject in the source images according to the motion in the driving video.
In our attention mechanism, dense similarities between the learned keypoints in the source and the driving images are computed.
To reduce the training-testing discrepancy of the self-supervised learning, a novel cross-identity training scheme is additionally introduced.
arXiv Detail & Related papers (2020-07-17T07:21:12Z) - First Order Motion Model for Image Animation [90.712718329677]
Image animation consists of generating a video sequence so that an object in a source image is animated according to the motion of a driving video.
Our framework addresses this problem without using any annotation or prior information about the specific object to animate.
arXiv Detail & Related papers (2020-02-29T07:08:56Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.