Streaming Video Diffusion: Online Video Editing with Diffusion Models

• URL: http://arxiv.org/abs/2405.19726v1
• Date: Thu, 30 May 2024 06:16:33 GMT
• Title: Streaming Video Diffusion: Online Video Editing with Diffusion Models
• Abstract summary: We present a novel task called online video editing, which is designed to edit textbfstreaming frames while maintaining temporal consistency. Our model can edit long, high-quality videos with remarkable results, achieving a real-time inference speed of 15.2 FPS at a resolution of 512x512.
• Score: 30.55613486291111
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a novel task called online video editing, which is designed to edit \textbf{streaming} frames while maintaining temporal consistency. Unlike existing offline video editing assuming all frames are pre-established and accessible, online video editing is tailored to real-life applications such as live streaming and online chat, requiring (1) fast continual step inference, (2) long-term temporal modeling, and (3) zero-shot video editing capability. To solve these issues, we propose Streaming Video Diffusion (SVDiff), which incorporates the compact spatial-aware temporal recurrence into off-the-shelf Stable Diffusion and is trained with the segment-level scheme on large-scale long videos. This simple yet effective setup allows us to obtain a single model that is capable of executing a broad range of videos and editing each streaming frame with temporal coherence. Our experiments indicate that our model can edit long, high-quality videos with remarkable results, achieving a real-time inference speed of 15.2 FPS at a resolution of 512x512.

Related papers

• EditaLive! Unified Character Video Editing for Live Streaming [59.93029261408041]
  EditaLive is a novel framework for real-time streaming character video editing.<n>It delivers state-of-the-art editing performance with faithful preservation of facial expressions and low-latency real-time streaming inference.
  arXiv  Detail & Related papers  (2026-08-27T13:37:45Z)
• EditStream: A Unified Autoregressive Framework for Interactive Video Generation and Editing [67.99022753510475]
  EditStream is a unified framework for interactive video generation and editing.<n>It unifies multiple video creation and manipulation tasks within a single DiT-based model.<n>It supports Text-to-Video, Image-to-Video, Video-to-Video, Editing propagation, Reference-guided Video Editing, and Camera Pose Change.
  arXiv  Detail & Related papers  (2026-08-16T07:27:18Z)
• OSVE: One Step Video Editing with One Step Diffusion Models [17.564785522861477]
  We present OSVE, the first framework to successfully adapt one-step Text-to-Image (T2I) models for high-quality video editing.<n>We address the core challenges of inversion, editability, and temporal consistency.<n>This breakthrough paves the way for practical, real-time video editing applications.
  arXiv  Detail & Related papers  (2026-07-22T08:29:30Z)
• LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editing [19.90420626930003]
  We present a novel streaming video editing framework that performs causal, frame-by-frame editing with strong content preservation and real-time responsiveness.<n>Our method achieves state-of-the-art visual quality among streaming baselines while drastically boosting inference speed to 12.66 FPS, making it suitable for interactive and augmented reality applications.
  arXiv  Detail & Related papers  (2026-06-25T08:24:03Z)
• ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformer [74.61793196579036]
  We propose a video-free tuning framework termed ViFeEdit for video diffusion transformers.<n>ViFeEdit achieves versatile video generation and editing, adapted solely with 2D images.<n>Our method delivers promising results of controllable video generation and editing with only minimal training on 2D image data.
  arXiv  Detail & Related papers  (2026-03-16T16:10:46Z)
• Memory-V2V: Augmenting Video-to-Video Diffusion Models with Memory [63.32726513381937]
  Current video editors struggle to maintain cross-consistency across sequential edits.<n>Memory-V2V is a framework that augments existing video-to-video models with explicit memory.<n>We show that Memory-V2V produces videos that are significantly more cross-consistent with minimal computational overhead.
  arXiv  Detail & Related papers  (2026-01-22T19:59:17Z)
• StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant [35.00546613484669]
  We present StreamBridge, a framework that seamlessly transforms offline Video-LLMs into streaming-capable models.<n>It addresses two fundamental challenges in adapting existing models into online scenarios: (1) limited capability for multi-turn real-time understanding, and (2) lack of proactive response mechanisms.
  arXiv  Detail & Related papers  (2025-05-08T17:57:40Z)
• Adaptive Caching for Faster Video Generation with Diffusion Transformers [52.73348147077075]
  Diffusion Transformers (DiTs) rely on larger models and heavier attention mechanisms, resulting in slower inference speeds. We introduce a training-free method to accelerate video DiTs, termed Adaptive Caching (AdaCache) We also introduce a Motion Regularization (MoReg) scheme to utilize video information within AdaCache, controlling the compute allocation based on motion content.
  arXiv  Detail & Related papers  (2024-11-04T18:59:44Z)
• Video-Infinity: Distributed Long Video Generation [73.30145218077074]
  Diffusion models have recently achieved remarkable results for video generation. Our method generates videos up to 2,300 frames in approximately 5 minutes, enabling long video generation at a speed 100 times faster than the prior methods.
  arXiv  Detail & Related papers  (2024-06-24T01:56:12Z)
• COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing [57.76170824395532]
  Video editing is an emerging task, in which most current methods adopt the pre-trained text-to-image (T2I) diffusion model to edit the source video. We propose COrrespondence-guided Video Editing (COVE) to achieve high-quality and consistent video editing. COVE can be seamlessly integrated into the pre-trained T2I diffusion model without the need for extra training or optimization.
  arXiv  Detail & Related papers  (2024-06-13T06:27:13Z)
• I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models [18.36472998650704]
  We introduce a novel and generic solution that extends the applicability of image editing tools to videos by propagating edits from a single frame to the entire video using a pre-trained image-to-video model. Our method, dubbed I2VEdit, adaptively preserves the visual and motion integrity of the source video depending on the extent of the edits.
  arXiv  Detail & Related papers  (2024-05-26T11:47:40Z)
• FastVideoEdit: Leveraging Consistency Models for Efficient Text-to-Video Editing [8.907836546058086]
  Existing approaches relying on image generation models for video editing suffer from time-consuming one-shot fine-tuning, additional condition extraction, or DDIM inversion. We propose FastVideoEdit, an efficient zero-shot video editing approach inspired by Consistency Models (CMs) Our method enables direct mapping from source video to target video with strong preservation ability utilizing a special variance schedule.
  arXiv  Detail & Related papers  (2024-03-10T17:12:01Z)
• RealCraft: Attention Control as A Tool for Zero-Shot Consistent Video Editing [10.356815854662862]
  We propose RealCraft, an attention-control-based method for zero-shot real-world video editing. By swapping cross-attention for new feature injection and relaxing spatial-temporal attention of the editing object, we achieve localized shape-wise edit. We showcase the proposed zero-shot attention-control-based method across a range of videos, demonstrating shape-wise, time-consistent and parameter-free editing.
  arXiv  Detail & Related papers  (2023-12-19T22:33:42Z)
• Neural Video Fields Editing [56.558490998753456]
  NVEdit is a text-driven video editing framework designed to mitigate memory overhead and improve consistency. We construct a neural video field, powered by tri-plane and sparse grid, to enable encoding long videos with hundreds of frames. Next, we update the video field through off-the-shelf Text-to-Image (T2I) models to text-driven editing effects.
  arXiv  Detail & Related papers  (2023-12-12T14:48:48Z)
• VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models [96.55004961251889]
  Video Instruction Diffusion (VIDiff) is a unified foundation model designed for a wide range of video tasks. Our model can edit and translate the desired results within seconds based on user instructions. We provide convincing generative results for diverse input videos and written instructions, both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2023-11-30T18:59:52Z)
• Edit Temporal-Consistent Videos with Image Diffusion Model [49.88186997567138]
  Large-scale text-to-image (T2I) diffusion models have been extended for text-guided video editing. T achieves state-of-the-art performance in both video temporal consistency and video editing capability.
  arXiv  Detail & Related papers  (2023-08-17T16:40:55Z)
• VidEdit: Zero-Shot and Spatially Aware Text-Driven Video Editing [18.24307442582304]
  We introduce VidEdit, a novel method for zero-shot text-based video editing. Our experiments show that VidEdit outperforms state-of-the-art methods on DAVIS dataset.
  arXiv  Detail & Related papers  (2023-06-14T19:15:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.