DNI: Dilutional Noise Initialization for Diffusion Video Editing

• URL: http://arxiv.org/abs/2409.13037v1
• Date: Thu, 19 Sep 2024 18:17:57 GMT
• Title: DNI: Dilutional Noise Initialization for Diffusion Video Editing
• Abstract summary: This paper proposes Dilutional Noise Initialization (DNI) framework which enables editing systems to perform precise and dynamic modification including non-rigid editing. DNI introduces a concept of noise dilution' which adds further noise to the latent noise in the region to be edited to soften the structural rigidity imposed by input video.
• Score: 22.308638156328968
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text-based diffusion video editing systems have been successful in performing edits with high fidelity and textual alignment. However, this success is limited to rigid-type editing such as style transfer and object overlay, while preserving the original structure of the input video. This limitation stems from an initial latent noise employed in diffusion video editing systems. The diffusion video editing systems prepare initial latent noise to edit by gradually infusing Gaussian noise onto the input video. However, we observed that the visual structure of the input video still persists within this initial latent noise, thereby restricting non-rigid editing such as motion change necessitating structural modifications. To this end, this paper proposes Dilutional Noise Initialization (DNI) framework which enables editing systems to perform precise and dynamic modification including non-rigid editing. DNI introduces a concept of `noise dilution' which adds further noise to the latent noise in the region to be edited to soften the structural rigidity imposed by input video, resulting in more effective edits closer to the target prompt. Extensive experiments demonstrate the effectiveness of the DNI framework.

Related papers

• Tuning-free Instruction-based Video Editing Via Structural Noise Initialization and Guidance [34.63499907054888]
  We propose a textittuning-free, instruction-based video editing framework.<n>Our proposed method achieves better visual quality and state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-05-15T02:09:06Z)
• FREE-Edit: Using Editing-aware Injection in Rectified Flow Models for Zero-shot Image-Driven Video Editing [12.549184989151135]
  Image-driven video editing aims to propagate edit contents from the modified first frame to the rest frames.<n>Current methods usually invert the source video to noise using a pre-trained image-to-video (I2V) model and then guide the sampling process using the edited first frame.<n>We propose an Editing-awaRE (REE) injection method to modulate injection intensity of each token.
  arXiv  Detail & Related papers  (2026-03-01T16:01:44Z)
• Region-Constraint In-Context Generation for Instructional Video Editing [91.27224696009755]
  We present ReCo, a new instructional video editing paradigm that delves into constraint modeling between editing and non-editing regions during in-context generation.<n>We propose a large-scale, high-quality video editing dataset, i.e., ReCo-Data, comprising 500K instruction-video pairs to benefit model training.
  arXiv  Detail & Related papers  (2025-12-19T14:49:30Z)
• Editable Noise Map Inversion: Encoding Target-image into Noise For High-Fidelity Image Manipulation [4.404496835736175]
  Key strategy for effective image editing involves inverting the source image into editable noise maps associated with the target image.<n>We propose Editable Noise Map Inversion (ENM Inversion), a novel inversion technique that searches for optimal noise maps to ensure both content preservation and editability.<n>Our approach can also be easily applied to video editing, enabling temporal consistency and content manipulation across frames.
  arXiv  Detail & Related papers  (2025-09-30T04:44:53Z)
• Taming Flow-based I2V Models for Creative Video Editing [64.67801702413122]
  Video editing, which aims to manipulate videos according to user intent, remains an emerging challenge.<n>Most existing image-conditioned video editing methods require inversion with model-specific design or need extensive optimization.<n>We propose IF-V2V, an Inversion-Free method that can adapt off-the-shelf flow-matching-based I2V models for video editing without significant computational overhead.
  arXiv  Detail & Related papers  (2025-09-26T05:57:04Z)
• Re-Attentional Controllable Video Diffusion Editing [48.052781838711994]
  We propose a Re-Attentional Controllable Video Diffusion Editing (ReAtCo) method. To align the spatial placement of the target objects with the edited text prompt in a training-free manner, we propose a Re-Attentional Diffusion (RAD) RAD refocuses the cross-attention activation responses between the edited text prompt and the target video during the denoising stage, resulting in a spatially location-aligned and semantically high-fidelity manipulated video.
  arXiv  Detail & Related papers  (2024-12-16T12:32:21Z)
• TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models [53.757752110493215]
  We focus on a popular line of text-based editing frameworks - the edit-friendly'' DDPM-noise inversion approach. We analyze its application to fast sampling methods and categorize its failures into two classes: the appearance of visual artifacts, and insufficient editing strength. We propose a pseudo-guidance approach that efficiently increases the magnitude of edits without introducing new artifacts.
  arXiv  Detail & Related papers  (2024-08-01T17:27:28Z)
• Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion [19.969947635371]
  Videoshop is a training-free video editing algorithm for localized semantic edits. It allows users to add or remove objects, semantically change objects, insert stock photos into videos, etc. with fine-grained control over locations and appearance. Videoshop produces higher quality edits against 6 baselines on 2 editing benchmarks using 10 evaluation metrics.
  arXiv  Detail & Related papers  (2024-03-21T17:59:03Z)
• UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing [28.140945021777878]
  We present UniEdit, a tuning-free framework that supports both video motion and appearance editing. To realize motion editing while preserving source video content, we introduce auxiliary motion-reference and reconstruction branches. The obtained features are then injected into the main editing path via temporal and spatial self-attention layers.
  arXiv  Detail & Related papers  (2024-02-20T17:52:12Z)
• Neutral Editing Framework for Diffusion-based Video Editing [24.370584544151424]
  This paper proposes Neutral Editing (NeuEdit) framework to enable complex non-rigid editing. NeuEdit introduces a concept of neutralization' that enhances a tuning-editing process of diffusion-based editing systems. Experiments on numerous videos demonstrate adaptability and effectiveness of the NeuEdit framework.
  arXiv  Detail & Related papers  (2023-12-10T16:28:32Z)
• Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training [61.984277261016146]
  We propose a CustomNeRF model that unifies a text description or a reference image as the editing prompt. To tackle the first challenge, we propose a Local-Global Iterative Editing (LGIE) training scheme that alternates between foreground region editing and full-image editing. For the second challenge, we also design a class-guided regularization that exploits class priors within the generation model to alleviate the inconsistency problem.
  arXiv  Detail & Related papers  (2023-12-04T06:25:06Z)
• MotionEditor: Editing Video Motion via Content-Aware Diffusion [96.825431998349]
  MotionEditor is a diffusion model for video motion editing. It incorporates a novel content-aware motion adapter into ControlNet to capture temporal motion correspondence.
  arXiv  Detail & Related papers  (2023-11-30T18:59:33Z)
• Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models [65.268245109828]
  Ground-A-Video is a video-to-video translation framework for multi-attribute video editing. It attains temporally consistent editing of input videos in a training-free manner. Experiments and applications demonstrate that Ground-A-Video's zero-shot capacity outperforms other baseline methods in terms of edit-accuracy and frame consistency.
  arXiv  Detail & Related papers  (2023-10-02T11:28:37Z)
• Editing 3D Scenes via Text Prompts without Retraining [80.57814031701744]
  DN2N is a text-driven editing method that allows for the direct acquisition of a NeRF model with universal editing capabilities. Our method employs off-the-shelf text-based editing models of 2D images to modify the 3D scene images. Our method achieves multiple editing types, including but not limited to appearance editing, weather transition, material changing, and style transfer.
  arXiv  Detail & Related papers  (2023-09-10T02:31:50Z)
• Edit-A-Video: Single Video Editing with Object-Aware Consistency [49.43316939996227]
  We propose a video editing framework given only a pretrained TTI model and a single text, video> pair, which we term Edit-A-Video. The framework consists of two stages: (1) inflating the 2D model into the 3D model by appending temporal modules tuning and on the source video (2) inverting the source video into the noise and editing with target text prompt and attention map injection. We present extensive experimental results over various types of text and videos, and demonstrate the superiority of the proposed method compared to baselines in terms of background consistency, text alignment, and video editing quality.
  arXiv  Detail & Related papers  (2023-03-14T14:35:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.