Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing

• URL: http://arxiv.org/abs/2405.04496v3
• Date: Tue, 15 Oct 2024 03:43:36 GMT
• Title: Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing
• Authors: Yi Zuo, Lingling Li, Licheng Jiao, Fang Liu, Xu Liu, Wenping Ma, Shuyuan Yang, Yuwei Guo,
• Abstract summary: We introduce Edit-Your-Motion, a video motion editing method that tackles unseen challenges through one-shot fine-tuning. To effectively decouple motion and appearance of source video, we design atemporal-two-stage learning strategy. With Edit-Your-Motion, users can edit the motion of humans in the source video, creating more engaging and diverse content.
• Score: 46.56615725175025
• License:
• Abstract: Existing diffusion-based methods have achieved impressive results in human motion editing. However, these methods often exhibit significant ghosting and body distortion in unseen in-the-wild cases. In this paper, we introduce Edit-Your-Motion, a video motion editing method that tackles these challenges through one-shot fine-tuning on unseen cases. Specifically, firstly, we utilized DDIM inversion to initialize the noise, preserving the appearance of the source video and designed a lightweight motion attention adapter module to enhance motion fidelity. DDIM inversion aims to obtain the implicit representations by estimating the prediction noise from the source video, which serves as a starting point for the sampling process, ensuring the appearance consistency between the source and edited videos. The Motion Attention Module (MA) enhances the model's motion editing ability by resolving the conflict between the skeleton features and the appearance features. Secondly, to effectively decouple motion and appearance of source video, we design a spatio-temporal two-stage learning strategy (STL). In the first stage, we focus on learning temporal features of human motion and propose recurrent causal attention (RCA) to ensure consistency between video frames. In the second stage, we shift focus on learning the appearance features of the source video. With Edit-Your-Motion, users can edit the motion of humans in the source video, creating more engaging and diverse content. Extensive qualitative and quantitative experiments, along with user preference studies, show that Edit-Your-Motion outperforms other methods.

Related papers

• MotionMatcher: Motion Customization of Text-to-Video Diffusion Models via Motion Feature Matching [27.28898943916193]
  Text-to-video (T2V) diffusion models have promising capabilities in synthesizing realistic videos from input text prompts. In this work, we tackle the motion customization problem, where a reference video is provided as motion guidance. We propose MotionMatcher, a motion customization framework that fine-tunes the pre-trained T2V diffusion model at the feature level.
  arXiv  Detail & Related papers  (2025-02-18T19:12:51Z)
• Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models [18.41701130228042]
  Motion customization aims to adapt the diffusion model (DM) to generate videos with the motion specified by a set of video clips with the same motion concept. This paper proposes two novel strategies to enhance motion-appearance separation, including temporal attention purification (TAP) and appearance highway (AH)
  arXiv  Detail & Related papers  (2025-01-28T05:40:20Z)
• Edit as You See: Image-guided Video Editing via Masked Motion Modeling [18.89936405508778]
  We propose a novel Image-guided Video Editing Diffusion model, termed IVEDiff. IVEDiff is built on top of image editing models, and is equipped with learnable motion modules to maintain the temporal consistency of edited video. Our method is able to generate temporally smooth edited videos while robustly dealing with various editing objects with high quality.
  arXiv  Detail & Related papers  (2025-01-08T07:52:12Z)
• MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models [59.10171699717122]
  MoTrans is a customized motion transfer method enabling video generation of similar motion in new context. multimodal representations from recaptioned prompt and video frames promote the modeling of appearance. Our method effectively learns specific motion pattern from singular or multiple reference videos.
  arXiv  Detail & Related papers  (2024-12-02T10:07:59Z)
• Zero-Shot Video Editing through Adaptive Sliding Score Distillation [51.57440923362033]
  This study proposes a novel paradigm of video-based score distillation, facilitating direct manipulation of original video content. We propose an Adaptive Sliding Score Distillation strategy, which incorporates both global and local video guidance to reduce the impact of editing errors.
  arXiv  Detail & Related papers  (2024-06-07T12:33:59Z)
• MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion [94.66090422753126]
  MotionFollower is a lightweight score-guided diffusion model for video motion editing. It delivers superior motion editing performance and exclusively supports large camera movements and actions. Compared with MotionEditor, the most advanced motion editing model, MotionFollower achieves an approximately 80% reduction in GPU memory.
  arXiv  Detail & Related papers  (2024-05-30T17:57:30Z)
• Animate Your Motion: Turning Still Images into Dynamic Videos [58.63109848837741]
  We introduce Scene and Motion Conditional Diffusion (SMCD), a novel methodology for managing multimodal inputs. SMCD incorporates a recognized motion conditioning module and investigates various approaches to integrate scene conditions. Our design significantly enhances video quality, motion precision, and semantic coherence.
  arXiv  Detail & Related papers  (2024-03-15T10:36:24Z)
• VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models [58.93124686141781]
  Video Motion Customization (VMC) is a novel one-shot tuning approach crafted to adapt temporal attention layers within video diffusion models. Our approach introduces a novel motion distillation objective using residual vectors between consecutive frames as a motion reference. We validate our method against state-of-the-art video generative models across diverse real-world motions and contexts.
  arXiv  Detail & Related papers  (2023-12-01T06:50:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.