InstructVid2Vid: Controllable Video Editing with Natural Language Instructions

• URL: http://arxiv.org/abs/2305.12328v2
• Date: Wed, 29 May 2024 11:08:41 GMT
• Title: InstructVid2Vid: Controllable Video Editing with Natural Language Instructions
• Authors: Bosheng Qin, Juncheng Li, Siliang Tang, Tat-Seng Chua, Yueting Zhuang,
• Abstract summary: InstructVid2Vid is an end-to-end diffusion-based methodology for video editing guided by human language instructions. Our approach empowers video manipulation guided by natural language directives, eliminating the need for per-example fine-tuning or inversion.
• Score: 97.17047888215284
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce InstructVid2Vid, an end-to-end diffusion-based methodology for video editing guided by human language instructions. Our approach empowers video manipulation guided by natural language directives, eliminating the need for per-example fine-tuning or inversion. The proposed InstructVid2Vid model modifies a pretrained image generation model, Stable Diffusion, to generate a time-dependent sequence of video frames. By harnessing the collective intelligence of disparate models, we engineer a training dataset rich in video-instruction triplets, which is a more cost-efficient alternative to collecting data in real-world scenarios. To enhance the coherence between successive frames within the generated videos, we propose the Inter-Frames Consistency Loss and incorporate it during the training process. With multimodal classifier-free guidance during the inference stage, the generated videos is able to resonate with both the input video and the accompanying instructions. Experimental results demonstrate that InstructVid2Vid is capable of generating high-quality, temporally coherent videos and performing diverse edits, including attribute editing, background changes, and style transfer. These results underscore the versatility and effectiveness of our proposed method.

Related papers

• Optical-Flow Guided Prompt Optimization for Coherent Video Generation [51.430833518070145]
  We propose a framework called MotionPrompt that guides the video generation process via optical flow. We optimize learnable token embeddings during reverse sampling steps by using gradients from a trained discriminator applied to random frame pairs. This approach allows our method to generate visually coherent video sequences that closely reflect natural motion dynamics, without compromising the fidelity of the generated content.
  arXiv  Detail & Related papers  (2024-11-23T12:26:52Z)
• Training-Free Semantic Video Composition via Pre-trained Diffusion Model [96.0168609879295]
  Current approaches, predominantly trained on videos with adjusted foreground color and lighting, struggle to address deep semantic disparities beyond superficial adjustments. We propose a training-free pipeline employing a pre-trained diffusion model imbued with semantic prior knowledge. Experimental results reveal that our pipeline successfully ensures the visual harmony and inter-frame coherence of the outputs.
  arXiv  Detail & Related papers  (2024-01-17T13:07:22Z)
• Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decoupling [79.49128866877922]
  Video-Teller is a video-language foundation model that leverages multi-modal fusion and fine-grained modality alignment. Video-Teller boosts the training efficiency by utilizing frozen pretrained vision and language modules. It capitalizes on the robust linguistic capabilities of large language models, enabling the generation of both concise and elaborate video descriptions.
  arXiv  Detail & Related papers  (2023-10-08T03:35:27Z)
• Animate-A-Story: Storytelling with Retrieval-Augmented Video Generation [69.20173154096]
  We develop a framework comprised of two functional modules, Motion Structure Retrieval and Structure-Guided Text-to-Video Synthesis. For the first module, we leverage an off-the-shelf video retrieval system and extract video depths as motion structure. For the second module, we propose a controllable video generation model that offers flexible controls over structure and characters.
  arXiv  Detail & Related papers  (2023-07-13T17:57:13Z)
• Structure and Content-Guided Video Synthesis with Diffusion Models [13.464501385061032]
  We present a structure and content-guided video diffusion model that edits videos based on visual or textual descriptions of the desired output. Our model is trained jointly on images and videos which also exposes explicit control of temporal consistency through a novel guidance method.
  arXiv  Detail & Related papers  (2023-02-06T18:50:23Z)
• TiVGAN: Text to Image to Video Generation with Step-by-Step Evolutionary Generator [34.7504057664375]
  We propose a novel training framework, Text-to-Image-to-Video Generative Adversarial Network (TiVGAN), which evolves frame-by-frame and finally produces a full-length video. Step-by-step learning process helps stabilize the training and enables the creation of high-resolution video based on conditional text descriptions.
  arXiv  Detail & Related papers  (2020-09-04T06:33:08Z)
• Unsupervised Multimodal Video-to-Video Translation via Self-Supervised Learning [92.17835753226333]
  We propose a novel unsupervised video-to-video translation model. Our model decomposes the style and the content using the specialized UV-decoder structure. Our model can produce photo-realistic videos in a multimodal way.
  arXiv  Detail & Related papers  (2020-04-14T13:44:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.