DragVideo: Interactive Drag-style Video Editing

• URL: http://arxiv.org/abs/2312.02216v3
• Date: Mon, 22 Jul 2024 05:25:28 GMT
• Title: DragVideo: Interactive Drag-style Video Editing
• Authors: Yufan Deng, Ruida Wang, Yuhao Zhang, Yu-Wing Tai, Chi-Keung Tang,
• Abstract summary: DragVideo is a general drag-temporal video editing framework. It can edit video in an intuitive, faithful to the user's intention manner, with nearly unnoticeable distortion and artifacts, while maintaining-temporal consistency. While traditional prompt-based video editing to do the former two and directly applying image drag editing fails in the last, DragVideo's versatility and generality are emphasized.
• Score: 58.59845960686982
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video generation models have shown their superior ability to generate photo-realistic video. However, how to accurately control (or edit) the video remains a formidable challenge. The main issues are: 1) how to perform direct and accurate user control in editing; 2) how to execute editings like changing shape, expression, and layout without unsightly distortion and artifacts to the edited content; and 3) how to maintain spatio-temporal consistency of video after editing. To address the above issues, we propose DragVideo, a general drag-style video editing framework. Inspired by DragGAN, DragVideo addresses issues 1) and 2) by proposing the drag-style video latent optimization method which gives desired control by updating noisy video latent according to drag instructions through video-level drag objective function. We amend issue 3) by integrating the video diffusion model with sample-specific LoRA and Mutual Self-Attention in DragVideo to ensure the edited result is spatio-temporally consistent. We also present a series of testing examples for drag-style video editing and conduct extensive experiments across a wide array of challenging editing tasks, such as motion, skeleton editing, etc, underscoring DragVideo can edit video in an intuitive, faithful to the user's intention manner, with nearly unnoticeable distortion and artifacts, while maintaining spatio-temporal consistency. While traditional prompt-based video editing fails to do the former two and directly applying image drag editing fails in the last, DragVideo's versatility and generality are emphasized. Github link: https://github.com/RickySkywalker/DragVideo-Official.

Related papers

• EasyV2V: A High-quality Instruction-based Video Editing Framework [108.78294392167017]
  captionemphEasyV2V is a framework for instruction-based video editing.<n>EasyV2V works with flexible inputs, e.g., video+text, video+mask+reference+, and state-of-the-art video editing results.
  arXiv  Detail & Related papers  (2025-12-18T18:59:57Z)
• MotionV2V: Editing Motion in a Video [53.791975554391534]
  We propose modifying video motion by editing sparse trajectories extracted from the input.<n>We term the deviation between input and output trajectories a "motion edit"<n>Our approach allows for edits that start at any timestamp and propagate naturally.
  arXiv  Detail & Related papers  (2025-11-25T18:57:25Z)
• Streaming Drag-Oriented Interactive Video Manipulation: Drag Anything, Anytime! [88.12304235156591]
  We propose textbfstReaming drag-oriEnted interactiVe vidEo manipuLation (REVEL), a new task that enables users to modify generated videos emphanytime on emphanything via fine-grained, interactive drag.<n>Our method can be seamlessly integrated into existing autoregressive video diffusion models.
  arXiv  Detail & Related papers  (2025-10-03T22:38:35Z)
• V2Edit: Versatile Video Diffusion Editor for Videos and 3D Scenes [29.80140472486948]
  V$2$Edit is a training-free framework for instruction-guided video and 3D scene editing. We introduce a progressive strategy that decomposes complex editing tasks into simpler subtasks. We extend V$2$Edit to 3D scene editing via a "render-edit-reconstruct" process, enabling high-quality, 3D-consistent edits.
  arXiv  Detail & Related papers  (2025-03-13T17:59:55Z)
• Drag Your Gaussian: Effective Drag-Based Editing with Score Distillation for 3D Gaussian Splatting [55.14822004410817]
  We introduce DYG, an effective 3D drag-based editing method for 3D Gaussian Splatting. It enables precise control over the extent of editing through the input of 3D masks and pairs of control points. DYG integrates the strengths of the implicit triplane representation to establish the geometric scaffold of the editing results.
  arXiv  Detail & Related papers  (2025-01-30T18:51:54Z)
• DragScene: Interactive 3D Scene Editing with Single-view Drag Instructions [9.31257776760014]
  3D editing has shown remarkable capability in editing scenes based on various instructions. Existing methods struggle with achieving intuitive, localized editing. We introduce DragScene, a framework that integrates drag-style editing with diverse 3D representations.
  arXiv  Detail & Related papers  (2024-12-18T07:02:01Z)
• I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models [18.36472998650704]
  We introduce a novel and generic solution that extends the applicability of image editing tools to videos by propagating edits from a single frame to the entire video using a pre-trained image-to-video model. Our method, dubbed I2VEdit, adaptively preserves the visual and motion integrity of the source video depending on the extent of the edits.
  arXiv  Detail & Related papers  (2024-05-26T11:47:40Z)
• ReVideo: Remake a Video with Motion and Content Control [67.5923127902463]
  We present a novel attempt to Remake a Video (VideoRe) which allows precise video editing in specific areas through the specification of both content and motion. VideoRe addresses a new task involving the coupling and training imbalance between content and motion control. Our method can also seamlessly extend these applications to multi-area editing without modifying specific training, demonstrating its flexibility and robustness.
  arXiv  Detail & Related papers  (2024-05-22T17:46:08Z)
• GenVideo: One-shot Target-image and Shape Aware Video Editing using T2I Diffusion Models [2.362412515574206]
  We propose "GenVideo" for editing videos leveraging target-image aware T2I models. Our approach handles edits with target objects of varying shapes and sizes while maintaining the temporal consistency of the edit.
  arXiv  Detail & Related papers  (2024-04-18T23:25:27Z)
• UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing [28.140945021777878]
  We present UniEdit, a tuning-free framework that supports both video motion and appearance editing. To realize motion editing while preserving source video content, we introduce auxiliary motion-reference and reconstruction branches. The obtained features are then injected into the main editing path via temporal and spatial self-attention layers.
  arXiv  Detail & Related papers  (2024-02-20T17:52:12Z)
• Neutral Editing Framework for Diffusion-based Video Editing [24.370584544151424]
  This paper proposes Neutral Editing (NeuEdit) framework to enable complex non-rigid editing. NeuEdit introduces a concept of neutralization' that enhances a tuning-editing process of diffusion-based editing systems. Experiments on numerous videos demonstrate adaptability and effectiveness of the NeuEdit framework.
  arXiv  Detail & Related papers  (2023-12-10T16:28:32Z)
• VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models [96.55004961251889]
  Video Instruction Diffusion (VIDiff) is a unified foundation model designed for a wide range of video tasks. Our model can edit and translate the desired results within seconds based on user instructions. We provide convincing generative results for diverse input videos and written instructions, both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2023-11-30T18:59:52Z)
• MotionEditor: Editing Video Motion via Content-Aware Diffusion [96.825431998349]
  MotionEditor is a diffusion model for video motion editing. It incorporates a novel content-aware motion adapter into ControlNet to capture temporal motion correspondence.
  arXiv  Detail & Related papers  (2023-11-30T18:59:33Z)
• Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models [65.268245109828]
  Ground-A-Video is a video-to-video translation framework for multi-attribute video editing. It attains temporally consistent editing of input videos in a training-free manner. Experiments and applications demonstrate that Ground-A-Video's zero-shot capacity outperforms other baseline methods in terms of edit-accuracy and frame consistency.
  arXiv  Detail & Related papers  (2023-10-02T11:28:37Z)
• MagicEdit: High-Fidelity and Temporally Coherent Video Editing [70.55750617502696]
  We present MagicEdit, a surprisingly simple yet effective solution to the text-guided video editing task. We found that high-fidelity and temporally coherent video-to-video translation can be achieved by explicitly disentangling the learning of content, structure and motion signals during training.
  arXiv  Detail & Related papers  (2023-08-28T17:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.