Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis

• URL: http://arxiv.org/abs/2507.13753v1
• Date: Fri, 18 Jul 2025 08:59:02 GMT
• Title: Encapsulated Composition of Text-to-Image and Text-to-Video Models for High-Quality Video Synthesis
• Authors: Tongtong Su, Chengyu Wang, Bingyan Liu, Jun Huang, Dongming Lu,
• Abstract summary: We introduce EVS, a training-free Encapsulated Video Synthesizer that composes T2I and T2V models to enhance both visual fidelity and motion smoothness.<n>Our approach utilizes a well-trained diffusion-based T2I model to refine low-quality video frames.<n>We also employ T2V backbones to ensure consistent motion dynamics.
• Score: 14.980220974022982
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, large text-to-video (T2V) synthesis models have garnered considerable attention for their abilities to generate videos from textual descriptions. However, achieving both high imaging quality and effective motion representation remains a significant challenge for these T2V models. Existing approaches often adapt pre-trained text-to-image (T2I) models to refine video frames, leading to issues such as flickering and artifacts due to inconsistencies across frames. In this paper, we introduce EVS, a training-free Encapsulated Video Synthesizer that composes T2I and T2V models to enhance both visual fidelity and motion smoothness of generated videos. Our approach utilizes a well-trained diffusion-based T2I model to refine low-quality video frames by treating them as out-of-distribution samples, effectively optimizing them with noising and denoising steps. Meanwhile, we employ T2V backbones to ensure consistent motion dynamics. By encapsulating the T2V temporal-only prior into the T2I generation process, EVS successfully leverages the strengths of both types of models, resulting in videos of improved imaging and motion quality. Experimental results validate the effectiveness of our approach compared to previous approaches. Our composition process also leads to a significant improvement of 1.6x-4.5x speedup in inference time. Source codes: https://github.com/Tonniia/EVS.

Related papers

• Enhancing Motion Dynamics of Image-to-Video Models via Adaptive Low-Pass Guidance [70.12690940725092]
  adaptive low-pass guidance (ALG) is a simple fix to the I2V model sampling procedure to generate more dynamic videos.<n>Under VBench-I2V test suite, ALG achieves an average improvement of 36% in dynamic degree without a significant drop in video quality or image fidelity.
  arXiv  Detail & Related papers  (2025-06-10T05:23:46Z)
• Incorporating Flexible Image Conditioning into Text-to-Video Diffusion Models without Training [27.794381157153776]
  We introduce a unified formulation for TI2V generation with flexible visual conditioning.<n>We propose an innovative training-free approach, dubbed FlexTI2V, that can condition T2V foundation models on an arbitrary amount of images.<n>Our method surpasses previous training-free image conditioning methods by a notable margin.
  arXiv  Detail & Related papers  (2025-05-27T02:16:06Z)
• Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model [133.01510927611452]
  We present Step-Video-T2V, a text-to-video pre-trained model with 30Bational parameters and the ability to generate videos up to 204 frames in length.<n>A deep compression Vari Autoencoder, Video-VAE, is designed for video generation tasks, achieving 16x16 spatial and 8x temporal compression ratios.<n>Step-Video-T2V's performance is evaluated on a novel video generation benchmark, Step-Video-T2V-Eval, demonstrating its state-of-the-art text-to-video quality.
  arXiv  Detail & Related papers  (2025-02-14T15:58:10Z)
• FrameBridge: Improving Image-to-Video Generation with Bridge Models [21.888786343816875]
  Diffusion models have achieved remarkable progress on image-to-video (I2V) generation.<n>Their noise-to-data generation process is inherently mismatched with this task, which may lead to suboptimal synthesis quality.<n>By modeling the frame-to-frames generation process with a bridge model based data-to-data generative process, we are able to fully exploit the information contained in the given image.
  arXiv  Detail & Related papers  (2024-10-20T12:10:24Z)
• VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide [48.22321420680046]
  VideoGuide is a novel framework that enhances the temporal consistency of pretrained text-to-video (T2V) models.<n>It improves temporal quality by interpolating the guiding model's denoised samples into the sampling model's denoising process.<n>The proposed method brings about significant improvement in temporal consistency and image fidelity.
  arXiv  Detail & Related papers  (2024-10-06T05:46:17Z)
• VideoElevator: Elevating Video Generation Quality with Versatile Text-to-Image Diffusion Models [94.25084162939488]
  Text-to-video diffusion models (T2V) still lag far behind in frame quality and text alignment. We introduce VideoElevator, a training-free and plug-and-play method, which elevates the performance of T2V using superior capabilities of T2I.
  arXiv  Detail & Related papers  (2024-03-08T16:44:54Z)
• LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models [133.088893990272]
  We learn a high-quality text-to-video (T2V) generative model by leveraging a pre-trained text-to-image (T2I) model as a basis. We propose LaVie, an integrated video generation framework that operates on cascaded video latent diffusion models.
  arXiv  Detail & Related papers  (2023-09-26T17:52:03Z)
• Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning [50.60891619269651]
  Control-A-Video is a controllable T2V diffusion model that can generate videos conditioned on text prompts and reference control maps like edge and depth maps. We propose novel strategies to incorporate content prior and motion prior into the diffusion-based generation process. Our framework generates higher-quality, more consistent videos compared to existing state-of-the-art methods in controllable text-to-video generation.
  arXiv  Detail & Related papers  (2023-05-23T09:03:19Z)
• Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation [31.882356164068753]
  To reproduce the success of text-to-image (T2I) generation, recent works in text-to-video (T2V) generation employ massive dataset for dataset for T2V generation. We propose Tune-A-Video is capable of producing temporally-coherent videos over various applications.
  arXiv  Detail & Related papers  (2022-12-22T09:43:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.