Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs

• URL: http://arxiv.org/abs/2308.13812v2
• Date: Tue, 19 Mar 2024 12:29:54 GMT
• Title: Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMs
• Authors: Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Tat-Seng Chua,
• Abstract summary: Text-to-video (T2V) synthesis has gained increasing attention in the community, in which the recently emerged diffusion models (DMs) have promisingly shown stronger performance than the past approaches. While existing state-of-the-art DMs are competent to achieve high-resolution video generation, they may largely suffer from key limitations (e.g., action occurrence disorders, crude video motions) with respect to the temporal dynamics modeling, one of the crux of video synthesis. In this work, we investigate strengthening awareness of video dynamics for DMs, for high-quality T2V generation
• Score: 112.39389727164594
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Text-to-video (T2V) synthesis has gained increasing attention in the community, in which the recently emerged diffusion models (DMs) have promisingly shown stronger performance than the past approaches. While existing state-of-the-art DMs are competent to achieve high-resolution video generation, they may largely suffer from key limitations (e.g., action occurrence disorders, crude video motions) with respect to the intricate temporal dynamics modeling, one of the crux of video synthesis. In this work, we investigate strengthening the awareness of video dynamics for DMs, for high-quality T2V generation. Inspired by human intuition, we design an innovative dynamic scene manager (dubbed as Dysen) module, which includes (step-1) extracting from input text the key actions with proper time-order arrangement, (step-2) transforming the action schedules into the dynamic scene graph (DSG) representations, and (step-3) enriching the scenes in the DSG with sufficient and reasonable details. Taking advantage of the existing powerful LLMs (e.g., ChatGPT) via in-context learning, Dysen realizes (nearly) human-level temporal dynamics understanding. Finally, the resulting video DSG with rich action scene details is encoded as fine-grained spatio-temporal features, integrated into the backbone T2V DM for video generating. Experiments on popular T2V datasets suggest that our Dysen-VDM consistently outperforms prior arts with significant margins, especially in scenarios with complex actions. Codes at https://haofei.vip/Dysen-VDM

Related papers

• DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training [85.04885553561164]
  Diffusion Transformers (DiTs) have shown remarkable performance in modeling and generating high-quality videos. This paper introduces DSV, a novel framework designed to accelerate and scale the training of video DiTs.
  arXiv  Detail & Related papers  (2025-02-11T14:39:59Z)
• Bridging Vision and Language: Modeling Causality and Temporality in Video Narratives [0.0]
  We propose an enhanced framework that integrates a Causal-Temporal Reasoning Module into state-of-the-art LVLMs. CTRM comprises two key components: the Causal Dynamics (CDE) and the Temporal Learner (TRL) We design a multi-stage learning strategy to optimize the model, combining pre-training on large-scale video-text datasets.
  arXiv  Detail & Related papers  (2024-12-14T07:28:38Z)
• Dynamic Try-On: Taming Video Virtual Try-on with Dynamic Attention Mechanism [52.9091817868613]
  Video try-on is a promising area for its tremendous real-world potential. Previous research has primarily focused on transferring product clothing images to videos with simple human poses. We propose a novel video try-on framework based on Diffusion Transformer(DiT), named Dynamic Try-On.
  arXiv  Detail & Related papers  (2024-12-13T03:20:53Z)
• VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval [8.908777234657046]
  Large-language and vision-language models (LLM/LVLMs) have gained prominence across various domains. Here we propose VideoLights, a novel HD/MR framework addressing these limitations through (i) Convolutional Projection and Feature Refinement modules. Comprehensive experiments on QVHighlights, TVSum, and Charades-STA benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-02T14:45:53Z)
• Individual Content and Motion Dynamics Preserved Pruning for Video Diffusion Models [26.556159722909715]
  High computational cost and slow inference time are major obstacles to deploying the video diffusion model (VDM) in practical applications. We introduce a new Video Diffusion Model Compression approach using individual content and motion dynamics preserved pruning and consistency loss.
  arXiv  Detail & Related papers  (2024-11-27T14:22:13Z)
• Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment [130.15775113897553]
  Finsta is a fine-grained structural-temporal alignment learning method. It consistently improves the existing 13 strong-tuning video-language models.
  arXiv  Detail & Related papers  (2024-06-27T15:23:36Z)
• VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis [18.806249040835624]
  We introduce the concept of Generative Temporal Nursing (GTN), where we aim to alter the generative process on the fly during inference to improve control over the temporal dynamics. We experimentally showcase the superiority of the proposed approach in generating longer, visually appealing videos over existing open-sourced T2V models.
  arXiv  Detail & Related papers  (2024-03-20T10:58:58Z)
• Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation [49.298187741014345]
  Current methods intertwine spatial content and temporal dynamics together, leading to an increased complexity of text-to-video generation (T2V) We propose HiGen, a diffusion model-based method that improves performance by decoupling the spatial and temporal factors of videos from two perspectives.
  arXiv  Detail & Related papers  (2023-12-07T17:59:07Z)
• Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models [71.11425812806431]
  Latent Diffusion Models (LDMs) enable high-quality image synthesis while avoiding excessive compute demands. Here, we apply the LDM paradigm to high-resolution generation, a particularly resource-intensive task. We focus on two relevant real-world applications: Simulation of in-the-wild driving data and creative content creation with text-to-video modeling.
  arXiv  Detail & Related papers  (2023-04-18T08:30:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.