UniVid: Pyramid Diffusion Model for High Quality Video Generation

• URL: http://arxiv.org/abs/2603.13739v1
• Date: Sat, 14 Mar 2026 03:51:16 GMT
• Title: UniVid: Pyramid Diffusion Model for High Quality Video Generation
• Abstract summary: Diffusion-based text-to-video generation (T2V) or image-to-video (I2V) generation have emerged as a prominent research focus.<n>We present a unified video generation model (UniVid) with hybrid conditions of the text prompt and reference image.
• Score: 10.349192062369088
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion-based text-to-video generation (T2V) or image-to-video (I2V) generation have emerged as a prominent research focus. However, there exists a challenge in integrating the two generative paradigms into a unified model. In this paper, we present a unified video generation model (UniVid) with hybrid conditions of the text prompt and reference image. Given these two available controls, our model can extract objects' appearance and their motion descriptions from textual prompts, while obtaining texture details and structural information from image clues to guide the video generation process. Specifically, we scale up the pre-trained text-to-image diffusion model for generating temporally coherent frames via introducing our temporal-pyramid cross-frame spatial-temporal attention modules and convolutions. To support bimodal control, we introduce a dual-stream cross-attention mechanism, whose attention scores can be freely re-weighted for interpolation of between single and two modalities controls during inference. Extensive experiments showcase that our UniVid achieves superior temporal coherence on T2V, I2V and (T+I)2V tasks.

Related papers

• Image-to-Video Diffusion: From Foundations to Open Frontiers [39.6216019326071]
  Diffusion-based textitimage-to-video (I2V) generation has become a central direction in generative models.<n>This work treats diffusion I2V generation as a standalone subject.<n>It first reviews the task formulation, model architectures, datasets, and evaluation metrics, and then organizes existing methods through a taxonomy based on architecture and training paradigm.
  arXiv  Detail & Related papers  (2026-05-17T04:10:55Z)
• BindWeave: Subject-Consistent Video Generation via Cross-Modal Integration [56.98981194478512]
  We propose a unified framework that handles a broad range of subject-to-video scenarios.<n>We introduce an MLLM-DiT framework in which a pretrained multimodal large language model performs deep cross-modal reasoning to ground entities.<n>Experiments on the OpenS2V benchmark demonstrate that our method achieves superior performance across subject consistency, naturalness, and text relevance in generated videos.
  arXiv  Detail & Related papers  (2025-10-01T02:41:11Z)
• FrameBridge: Improving Image-to-Video Generation with Bridge Models [21.888786343816875]
  Diffusion models have achieved remarkable progress on image-to-video (I2V) generation.<n>Their noise-to-data generation process is inherently mismatched with this task, which may lead to suboptimal synthesis quality.<n>By modeling the frame-to-frames generation process with a bridge model based data-to-data generative process, we are able to fully exploit the information contained in the given image.
  arXiv  Detail & Related papers  (2024-10-20T12:10:24Z)
• FancyVideo: Towards Dynamic and Consistent Video Generation via Cross-frame Textual Guidance [3.6519202494141125]
  We introduce FancyVideo, an innovative video generator that improves the existing text-control mechanism.<n>CTGM incorporates the Temporal Information (TII) and Temporal Affinity Refiner (TAR) at the beginning and end of cross-attention.<n>Our approach achieves state-of-the-art T2V generation results on the EvalCrafter benchmark.
  arXiv  Detail & Related papers  (2024-08-15T14:47:44Z)
• Vivid-ZOO: Multi-View Video Generation with Diffusion Model [76.96449336578286]
  New challenges lie in the lack of massive captioned multi-view videos and the complexity of modeling such multi-dimensional distribution. We propose a novel diffusion-based pipeline that generates high-quality multi-view videos centered around a dynamic 3D object from text.
  arXiv  Detail & Related papers  (2024-06-12T21:44:04Z)
• S2DM: Sector-Shaped Diffusion Models for Video Generation [2.0270353391739637]
  We propose a novel Sector-Shaped Diffusion Model (S2DM) for video generation. S2DM can generate a group of intrinsically related data sharing the same semantic and intrinsically related features. We show that, without additional training, our model integrated with another temporal conditions generative model can still achieve comparable performance with existing works.
  arXiv  Detail & Related papers  (2024-03-20T08:50:15Z)
• Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation [72.90144343056227]
  We explore the visual representations produced from a pre-trained text-to-video (T2V) diffusion model for video understanding tasks. We introduce a novel framework, termed "VD-IT", tailored with dedicatedly designed components built upon a fixed T2V model. Our VD-IT achieves highly competitive results, surpassing many existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-03-18T17:59:58Z)
• I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models [80.32562822058924]
  Text-guided image-to-video (I2V) generation aims to generate a coherent video that preserves the identity of the input image. I2V-Adapter adeptly propagates the unnoised input image to subsequent noised frames through a cross-frame attention mechanism. Our experimental results demonstrate that I2V-Adapter is capable of producing high-quality videos.
  arXiv  Detail & Related papers  (2023-12-27T19:11:50Z)
• Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation [49.298187741014345]
  Current methods intertwine spatial content and temporal dynamics together, leading to an increased complexity of text-to-video generation (T2V) We propose HiGen, a diffusion model-based method that improves performance by decoupling the spatial and temporal factors of videos from two perspectives.
  arXiv  Detail & Related papers  (2023-12-07T17:59:07Z)
• Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation [55.36617538438858]
  We propose a novel approach that strengthens the interaction between spatial and temporal perceptions. We curate a large-scale and open-source video dataset called HD-VG-130M.
  arXiv  Detail & Related papers  (2023-05-18T11:06:15Z)
• Make-A-Video: Text-to-Video Generation without Text-Video Data [69.20996352229422]
  Make-A-Video is an approach for translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V) We design a simple yet effective way to build on T2I models with novel and effective spatial-temporal modules. In all aspects, spatial and temporal resolution, faithfulness to text, and quality, Make-A-Video sets the new state-of-the-art in text-to-video generation.
  arXiv  Detail & Related papers  (2022-09-29T13:59:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.