GenVideo: One-shot Target-image and Shape Aware Video Editing using T2I Diffusion Models

• URL: http://arxiv.org/abs/2404.12541v1
• Date: Thu, 18 Apr 2024 23:25:27 GMT
• Title: GenVideo: One-shot Target-image and Shape Aware Video Editing using T2I Diffusion Models
• Authors: Sai Sree Harsha, Ambareesh Revanur, Dhwanit Agarwal, Shradha Agrawal,
• Abstract summary: We propose "GenVideo" for editing videos leveraging target-image aware T2I models. Our approach handles edits with target objects of varying shapes and sizes while maintaining the temporal consistency of the edit.
• Score: 2.362412515574206
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Video editing methods based on diffusion models that rely solely on a text prompt for the edit are hindered by the limited expressive power of text prompts. Thus, incorporating a reference target image as a visual guide becomes desirable for precise control over edit. Also, most existing methods struggle to accurately edit a video when the shape and size of the object in the target image differ from the source object. To address these challenges, we propose "GenVideo" for editing videos leveraging target-image aware T2I models. Our approach handles edits with target objects of varying shapes and sizes while maintaining the temporal consistency of the edit using our novel target and shape aware InvEdit masks. Further, we propose a novel target-image aware latent noise correction strategy during inference to improve the temporal consistency of the edits. Experimental analyses indicate that GenVideo can effectively handle edits with objects of varying shapes, where existing approaches fail.

Related papers

• A Survey of Multimodal-Guided Image Editing with Text-to-Image Diffusion Models [117.77807994397784]
  Image editing aims to edit the given synthetic or real image to meet the specific requirements from users. Recent significant advancement in this field is based on the development of text-to-image (T2I) diffusion models. T2I-based image editing methods significantly enhance editing performance and offer a user-friendly interface for modifying content guided by multimodal inputs.
  arXiv  Detail & Related papers  (2024-06-20T17:58:52Z)
• Temporally Consistent Object Editing in Videos using Extended Attention [9.605596668263173]
  We propose a method to edit videos using a pre-trained inpainting image diffusion model. We ensure that the edited information will be consistent across all the video frames.
  arXiv  Detail & Related papers  (2024-06-01T02:31:16Z)
• I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models [18.36472998650704]
  We introduce a novel and generic solution that extends the applicability of image editing tools to videos by propagating edits from a single frame to the entire video using a pre-trained image-to-video model. Our method, dubbed I2VEdit, adaptively preserves the visual and motion integrity of the source video depending on the extent of the edits.
  arXiv  Detail & Related papers  (2024-05-26T11:47:40Z)
• Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion [19.969947635371]
  Videoshop is a training-free video editing algorithm for localized semantic edits. It allows users to add or remove objects, semantically change objects, insert stock photos into videos, etc. with fine-grained control over locations and appearance. Videoshop produces higher quality edits against 6 baselines on 2 editing benchmarks using 10 evaluation metrics.
  arXiv  Detail & Related papers  (2024-03-21T17:59:03Z)
• Customize your NeRF: Adaptive Source Driven 3D Scene Editing via Local-Global Iterative Training [61.984277261016146]
  We propose a CustomNeRF model that unifies a text description or a reference image as the editing prompt. To tackle the first challenge, we propose a Local-Global Iterative Editing (LGIE) training scheme that alternates between foreground region editing and full-image editing. For the second challenge, we also design a class-guided regularization that exploits class priors within the generation model to alleviate the inconsistency problem.
  arXiv  Detail & Related papers  (2023-12-04T06:25:06Z)
• Object-aware Inversion and Reassembly for Image Editing [61.19822563737121]
  We propose Object-aware Inversion and Reassembly (OIR) to enable object-level fine-grained editing. We use our search metric to find the optimal inversion step for each editing pair when editing an image. Our method achieves superior performance in editing object shapes, colors, materials, categories, etc., especially in multi-object editing scenarios.
  arXiv  Detail & Related papers  (2023-10-18T17:59:02Z)
• Editing 3D Scenes via Text Prompts without Retraining [80.57814031701744]
  DN2N is a text-driven editing method that allows for the direct acquisition of a NeRF model with universal editing capabilities. Our method employs off-the-shelf text-based editing models of 2D images to modify the 3D scene images. Our method achieves multiple editing types, including but not limited to appearance editing, weather transition, material changing, and style transfer.
  arXiv  Detail & Related papers  (2023-09-10T02:31:50Z)
• InFusion: Inject and Attention Fusion for Multi Concept Zero-Shot Text-based Video Editing [27.661609140918916]
  InFusion is a framework for zero-shot text-based video editing. It supports editing of multiple concepts with pixel-level control over diverse concepts mentioned in the editing prompt. Our framework is a low-cost alternative to one-shot tuned models for editing since it does not require training.
  arXiv  Detail & Related papers  (2023-07-22T17:05:47Z)
• Edit-A-Video: Single Video Editing with Object-Aware Consistency [49.43316939996227]
  We propose a video editing framework given only a pretrained TTI model and a single text, video> pair, which we term Edit-A-Video. The framework consists of two stages: (1) inflating the 2D model into the 3D model by appending temporal modules tuning and on the source video (2) inverting the source video into the noise and editing with target text prompt and attention map injection. We present extensive experimental results over various types of text and videos, and demonstrate the superiority of the proposed method compared to baselines in terms of background consistency, text alignment, and video editing quality.
  arXiv  Detail & Related papers  (2023-03-14T14:35:59Z)
• Video-P2P: Video Editing with Cross-attention Control [68.64804243427756]
  Video-P2P is a novel framework for real-world video editing with cross-attention control. Video-P2P works well on real-world videos for generating new characters while optimally preserving their original poses and scenes.
  arXiv  Detail & Related papers  (2023-03-08T17:53:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.