VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing

• URL: http://arxiv.org/abs/2502.17258v1
• Date: Mon, 24 Feb 2025 15:39:14 GMT
• Title: VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing
• Authors: Xiangpeng Yang, Linchao Zhu, Hehe Fan, Yi Yang,
• Abstract summary: VideoGrain is a zero-shot approach that modulates space-time to achieve fine-grained control over video content.<n>We enhance text-to-region control by amplifying each local prompt's attention to its corresponding spatial-disentangled region.<n>We improve feature separation by increasing intra-region awareness and reducing inter-region interference in self-attention.
• Score: 62.15822650722473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in diffusion models have significantly improved video generation and editing capabilities. However, multi-grained video editing, which encompasses class-level, instance-level, and part-level modifications, remains a formidable challenge. The major difficulties in multi-grained editing include semantic misalignment of text-to-region control and feature coupling within the diffusion model. To address these difficulties, we present VideoGrain, a zero-shot approach that modulates space-time (cross- and self-) attention mechanisms to achieve fine-grained control over video content. We enhance text-to-region control by amplifying each local prompt's attention to its corresponding spatial-disentangled region while minimizing interactions with irrelevant areas in cross-attention. Additionally, we improve feature separation by increasing intra-region awareness and reducing inter-region interference in self-attention. Extensive experiments demonstrate our method achieves state-of-the-art performance in real-world scenarios. Our code, data, and demos are available at https://knightyxp.github.io/VideoGrain_project_page/

Related papers

• MAKIMA: Tuning-free Multi-Attribute Open-domain Video Editing via Mask-Guided Attention Modulation [55.101611012677616]
  Diffusion-based text-to-image (T2I) models have demonstrated remarkable results in global video editing tasks.<n>We present MAKIMA, a tuning-free MAE framework built upon pretrained T2I models for open-domain video editing.
  arXiv  Detail & Related papers  (2024-12-28T02:36:51Z)
• Re-Attentional Controllable Video Diffusion Editing [48.052781838711994]
  We propose a Re-Attentional Controllable Video Diffusion Editing (ReAtCo) method.<n>To align the spatial placement of the target objects with the edited text prompt in a training-free manner, we propose a Re-Attentional Diffusion (RAD)<n>RAD refocuses the cross-attention activation responses between the edited text prompt and the target video during the denoising stage, resulting in a spatially location-aligned and semantically high-fidelity manipulated video.
  arXiv  Detail & Related papers  (2024-12-16T12:32:21Z)
• A dual contrastive framework [7.358205057611624]
  Region-level visual understanding presents significant challenges for large-scale vision-language models.<n>We propose AlignCap, a framework designed to enhance region-level understanding through fine-grained alignment of latent spaces.
  arXiv  Detail & Related papers  (2024-12-13T18:45:18Z)
• COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing [57.76170824395532]
  Video editing is an emerging task, in which most current methods adopt the pre-trained text-to-image (T2I) diffusion model to edit the source video.<n>We propose COrrespondence-guided Video Editing (COVE) to achieve high-quality and consistent video editing.<n>COVE can be seamlessly integrated into the pre-trained T2I diffusion model without the need for extra training or optimization.
  arXiv  Detail & Related papers  (2024-06-13T06:27:13Z)
• Temporally Consistent Object Editing in Videos using Extended Attention [9.605596668263173]
  We propose a method to edit videos using a pre-trained inpainting image diffusion model. We ensure that the edited information will be consistent across all the video frames.
  arXiv  Detail & Related papers  (2024-06-01T02:31:16Z)
• Multi-scale 2D Temporal Map Diffusion Models for Natural Language Video Localization [85.85582751254785]
  We present a novel approach to NLVL that aims to address this issue. Our method involves the direct generation of a global 2D temporal map via a conditional denoising diffusion process. Our approach effectively encapsulates the interaction between the query and video data across various time scales.
  arXiv  Detail & Related papers  (2024-01-16T09:33:29Z)
• Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models [65.268245109828]
  Ground-A-Video is a video-to-video translation framework for multi-attribute video editing. It attains temporally consistent editing of input videos in a training-free manner. Experiments and applications demonstrate that Ground-A-Video's zero-shot capacity outperforms other baseline methods in terms of edit-accuracy and frame consistency.
  arXiv  Detail & Related papers  (2023-10-02T11:28:37Z)
• Video Region Annotation with Sparse Bounding Boxes [29.323784279321337]
  We learn to automatically generate region boundaries for all frames of a video from sparsely annotated bounding boxes of target regions. We achieve this with a Volumetric Graph Convolutional Network (VGCN), which learns to iteratively find keypoints on the region boundaries.
  arXiv  Detail & Related papers  (2020-08-17T01:27:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.