Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing

• URL: http://arxiv.org/abs/2410.12526v1
• Date: Wed, 16 Oct 2024 13:03:15 GMT
• Title: Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing
• Authors: Mingce Guo, Jingxuan He, Shengeng Tang, Zhangye Wang, Lechao Cheng,
• Abstract summary: This work proposes an improved concept-augmented video editing approach that generates diverse and stable target videos flexibly. The framework involves concept-augmented textual inversion and a dual prior supervision mechanism. Comprehensive evaluations demonstrate that our approach generates more stable and lifelike videos, outperforming state-of-the-art methods.
• Score: 12.38953947065143
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-driven video editing utilizing generative diffusion models has garnered significant attention due to their potential applications. However, existing approaches are constrained by the limited word embeddings provided in pre-training, which hinders nuanced editing targeting open concepts with specific attributes. Directly altering the keywords in target prompts often results in unintended disruptions to the attention mechanisms. To achieve more flexible editing easily, this work proposes an improved concept-augmented video editing approach that generates diverse and stable target videos flexibly by devising abstract conceptual pairs. Specifically, the framework involves concept-augmented textual inversion and a dual prior supervision mechanism. The former enables plug-and-play guidance of stable diffusion for video editing, effectively capturing target attributes for more stylized results. The dual prior supervision mechanism significantly enhances video stability and fidelity. Comprehensive evaluations demonstrate that our approach generates more stable and lifelike videos, outperforming state-of-the-art methods.

Related papers

• Beyond Generation: Unlocking Universal Editing via Self-Supervised Fine-Tuning [45.64777118760738]
  UES (Unlocking Universal Editing via Self-Supervision) is a lightweight self-supervised fine-tuning strategy that transforms generation models into unified generation-editing systems. Our approach establishes a dual-conditioning mechanism where original video-text pairs jointly provide visual and textual semantics. To enable systematic evaluation, we introduce OmniBench-99, a comprehensive benchmark spanning 99 videos across humans/animals, environments, and objects.
  arXiv  Detail & Related papers  (2024-12-03T03:10:19Z)
• Zero-Shot Video Editing through Adaptive Sliding Score Distillation [51.57440923362033]
  This study proposes a novel paradigm of video-based score distillation, facilitating direct manipulation of original video content. We propose an Adaptive Sliding Score Distillation strategy, which incorporates both global and local video guidance to reduce the impact of editing errors.
  arXiv  Detail & Related papers  (2024-06-07T12:33:59Z)
• I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models [18.36472998650704]
  We introduce a novel and generic solution that extends the applicability of image editing tools to videos by propagating edits from a single frame to the entire video using a pre-trained image-to-video model. Our method, dubbed I2VEdit, adaptively preserves the visual and motion integrity of the source video depending on the extent of the edits.
  arXiv  Detail & Related papers  (2024-05-26T11:47:40Z)
• ReVideo: Remake a Video with Motion and Content Control [67.5923127902463]
  We present a novel attempt to Remake a Video (VideoRe) which allows precise video editing in specific areas through the specification of both content and motion. VideoRe addresses a new task involving the coupling and training imbalance between content and motion control. Our method can also seamlessly extend these applications to multi-area editing without modifying specific training, demonstrating its flexibility and robustness.
  arXiv  Detail & Related papers  (2024-05-22T17:46:08Z)
• DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editing [48.238213651343784]
  Video score distillation can introduce new content indicated by target text, but can also cause structure and motion deviation. We propose to match space-time self-similarities of the original video and the edited video during the score distillation. Our approach is model-agnostic, which can be applied for both cascaded and non-cascaded video diffusion frameworks.
  arXiv  Detail & Related papers  (2024-03-18T17:38:53Z)
• StableVideo: Text-driven Consistency-aware Diffusion Video Editing [24.50933856309234]
  Diffusion-based methods can generate realistic images and videos, but they struggle to edit existing objects in a video while preserving their appearance over time. This paper introduces temporal dependency to existing text-driven diffusion models, which allows them to generate consistent appearance for the edited objects. We build up a text-driven video editing framework based on this mechanism, namely StableVideo, which can achieve consistency-aware video editing.
  arXiv  Detail & Related papers  (2023-08-18T14:39:16Z)
• InstructVid2Vid: Controllable Video Editing with Natural Language Instructions [97.17047888215284]
  InstructVid2Vid is an end-to-end diffusion-based methodology for video editing guided by human language instructions. Our approach empowers video manipulation guided by natural language directives, eliminating the need for per-example fine-tuning or inversion.
  arXiv  Detail & Related papers  (2023-05-21T03:28:13Z)
• Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts [116.05656635044357]
  We propose a generic video editing framework called Make-A-Protagonist. Specifically, we leverage multiple experts to parse source video, target visual and textual clues, and propose a visual-textual-based video generation model. Results demonstrate the versatile and remarkable editing capabilities of Make-A-Protagonist.
  arXiv  Detail & Related papers  (2023-05-15T17:59:03Z)
• Video-P2P: Video Editing with Cross-attention Control [68.64804243427756]
  Video-P2P is a novel framework for real-world video editing with cross-attention control. Video-P2P works well on real-world videos for generating new characters while optimally preserving their original poses and scenes.
  arXiv  Detail & Related papers  (2023-03-08T17:53:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.