Motion-Conditioned Image Animation for Video Editing
- URL: http://arxiv.org/abs/2311.18827v1
- Date: Thu, 30 Nov 2023 18:59:06 GMT
- Title: Motion-Conditioned Image Animation for Video Editing
- Authors: Wilson Yan, Andrew Brown, Pieter Abbeel, Rohit Girdhar, Samaneh Azadi
- Abstract summary: MoCA is a Motion-Conditioned Image Animation approach for video editing.
We present a comprehensive human evaluation of the latest video editing methods along with MoCA, on our proposed benchmark.
MoCA establishes a new state-of-the-art, demonstrating greater human preference win-rate, and outperforming notable recent approaches.
- Score: 65.90398261600964
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce MoCA, a Motion-Conditioned Image Animation approach for video
editing. It leverages a simple decomposition of the video editing problem into
image editing followed by motion-conditioned image animation. Furthermore,
given the lack of robust evaluation datasets for video editing, we introduce a
new benchmark that measures edit capability across a wide variety of tasks,
such as object replacement, background changes, style changes, and motion
edits. We present a comprehensive human evaluation of the latest video editing
methods along with MoCA, on our proposed benchmark. MoCA establishes a new
state-of-the-art, demonstrating greater human preference win-rate, and
outperforming notable recent approaches including Dreamix (63%), MasaCtrl
(75%), and Tune-A-Video (72%), with especially significant improvements for
motion edits.
Related papers
- MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion [94.66090422753126]
MotionFollower is a lightweight score-guided diffusion model for video motion editing.
It delivers superior motion editing performance and exclusively supports large camera movements and actions.
Compared with MotionEditor, the most advanced motion editing model, MotionFollower achieves an approximately 80% reduction in GPU memory.
arXiv Detail & Related papers (2024-05-30T17:57:30Z) - I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models [18.36472998650704]
We introduce a novel and generic solution that extends the applicability of image editing tools to videos by propagating edits from a single frame to the entire video using a pre-trained image-to-video model.
Our method, dubbed I2VEdit, adaptively preserves the visual and motion integrity of the source video depending on the extent of the edits.
arXiv Detail & Related papers (2024-05-26T11:47:40Z) - ReVideo: Remake a Video with Motion and Content Control [67.5923127902463]
We present a novel attempt to Remake a Video (VideoRe) which allows precise video editing in specific areas through the specification of both content and motion.
VideoRe addresses a new task involving the coupling and training imbalance between content and motion control.
Our method can also seamlessly extend these applications to multi-area editing without modifying specific training, demonstrating its flexibility and robustness.
arXiv Detail & Related papers (2024-05-22T17:46:08Z) - Action Reimagined: Text-to-Pose Video Editing for Dynamic Human Actions [49.14827857853878]
ReimaginedAct comprises video understanding, reasoning, and editing modules.
Our method can accept not only direct instructional text prompts but also what if' questions to predict possible action changes.
arXiv Detail & Related papers (2024-03-11T22:46:46Z) - SAVE: Protagonist Diversification with Structure Agnostic Video Editing [29.693364686494274]
Previous works usually work well on trivial and consistent shapes, and easily collapse on a difficult target that has a largely different body shape from the original one.
We propose motion personalization that isolates the motion from a single source video and then modifies the protagonist accordingly.
We also regulate the motion word to attend to proper motion-related areas by introducing a novel pseudo optical flow.
arXiv Detail & Related papers (2023-12-05T05:13:20Z) - MotionEditor: Editing Video Motion via Content-Aware Diffusion [96.825431998349]
MotionEditor is a diffusion model for video motion editing.
It incorporates a novel content-aware motion adapter into ControlNet to capture temporal motion correspondence.
arXiv Detail & Related papers (2023-11-30T18:59:33Z) - MagicProp: Diffusion-based Video Editing via Motion-aware Appearance
Propagation [74.32046206403177]
MagicProp disentangles the video editing process into two stages: appearance editing and motion-aware appearance propagation.
In the first stage, MagicProp selects a single frame from the input video and applies image-editing techniques to modify the content and/or style of the frame.
In the second stage, MagicProp employs the edited frame as an appearance reference and generates the remaining frames using an autoregressive rendering approach.
arXiv Detail & Related papers (2023-09-02T11:13:29Z) - Dreamix: Video Diffusion Models are General Video Editors [22.127604561922897]
Text-driven image and video diffusion models have recently achieved unprecedented generation realism.
We present the first diffusion-based method that is able to perform text-based motion and appearance editing of general videos.
arXiv Detail & Related papers (2023-02-02T18:58:58Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.