UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video Super-Resolution

• URL: http://arxiv.org/abs/2510.08143v1
• Date: Thu, 09 Oct 2025 12:25:16 GMT
• Title: UniMMVSR: A Unified Multi-Modal Framework for Cascaded Video Super-Resolution
• Authors: Shian Du, Menghan Xia, Chang Liu, Quande Liu, Xintao Wang, Pengfei Wan, Xiangyang Ji,
• Abstract summary: Cascaded video super-resolution has emerged as a promising technique for generating high-resolution videos using large foundation models.<n>We present UniMMVSR, the first unified generative video super-resolution framework to incorporate hybrid-modal conditions, including text, images, and videos.<n>Our experiments demonstrate that UniMMVSR significantly outperforms existing methods, producing videos with superior detail and a higher degree of conformity to multi-modal conditions.
• Score: 62.10676832966289
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Cascaded video super-resolution has emerged as a promising technique for decoupling the computational burden associated with generating high-resolution videos using large foundation models. Existing studies, however, are largely confined to text-to-video tasks and fail to leverage additional generative conditions beyond text, which are crucial for ensuring fidelity in multi-modal video generation. We address this limitation by presenting UniMMVSR, the first unified generative video super-resolution framework to incorporate hybrid-modal conditions, including text, images, and videos. We conduct a comprehensive exploration of condition injection strategies, training schemes, and data mixture techniques within a latent video diffusion model. A key challenge was designing distinct data construction and condition utilization methods to enable the model to precisely utilize all condition types, given their varied correlations with the target video. Our experiments demonstrate that UniMMVSR significantly outperforms existing methods, producing videos with superior detail and a higher degree of conformity to multi-modal conditions. We also validate the feasibility of combining UniMMVSR with a base model to achieve multi-modal guided generation of 4K video, a feat previously unattainable with existing techniques.

Related papers

• VDOT: Efficient Unified Video Creation via Optimal Transport Distillation [70.02065520468726]
  We propose an efficient unified video creation model, named VDOT.<n>We employ a novel computational optimal transport (OT) technique to optimize the discrepancy between the real and fake score distributions.<n>To support training unified video creation models, we propose a fully automated pipeline for video data annotation and filtering.
  arXiv  Detail & Related papers  (2025-12-07T11:31:00Z)
• Team of One: Cracking Complex Video QA with Model Synergy [24.75732964829523]
  We propose a novel framework for open-ended video question answering that enhances reasoning depth and robustness in complex real-world scenarios.<n>Existing Video-Large Multimodal Models (Video-LMMs) often exhibit limited contextual understanding, weak temporal modeling, and poor generalization to ambiguous or compositional queries.
  arXiv  Detail & Related papers  (2025-07-18T11:12:44Z)
• MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglement [47.064467920954776]
  We introduce MAGREF, a unified and effective framework for any-reference video generation.<n>Our approach incorporates masked guidance and a subject disentanglement mechanism.<n>Experiments on a comprehensive benchmark demonstrate that MAGREF consistently outperforms existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-05-29T17:58:15Z)
• Redefining Temporal Modeling in Video Diffusion: The Vectorized Timestep Approach [29.753974393652356]
  We propose a frame-aware video diffusion model(FVDM) Our approach allows each frame to follow an independent noise schedule, enhancing the model's capacity to capture fine-grained temporal dependencies. Our empirical evaluations show that FVDM outperforms state-of-the-art methods in video generation quality, while also excelling in extended tasks.
  arXiv  Detail & Related papers  (2024-10-04T05:47:39Z)
• VIMI: Grounding Video Generation through Multi-modal Instruction [89.90065445082442]
  Existing text-to-video diffusion models rely solely on text-only encoders for their pretraining. We construct a large-scale multimodal prompt dataset by employing retrieval methods to pair in-context examples with the given text prompts. We finetune the model from the first stage on three video generation tasks, incorporating multi-modal instructions.
  arXiv  Detail & Related papers  (2024-07-08T18:12:49Z)
• Video Interpolation with Diffusion Models [54.06746595879689]
  We present VIDIM, a generative model for video, which creates short videos given a start and end frame. VIDIM uses cascaded diffusion models to first generate the target video at low resolution, and then generate the high-resolution video conditioned on the low-resolution generated video.
  arXiv  Detail & Related papers  (2024-04-01T15:59:32Z)
• Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets [36.95521842177614]
  We present Stable Video Diffusion - a latent video diffusion model for high-resolution, state-of-the-art text-to-video and image-to-video generation. We identify and evaluate three different stages for successful training of video LDMs: text-to-image pretraining, video pretraining, and high-quality video finetuning.
  arXiv  Detail & Related papers  (2023-11-25T22:28:38Z)
• Video Probabilistic Diffusion Models in Projected Latent Space [75.4253202574722]
  We propose a novel generative model for videos, coined projected latent video diffusion models (PVDM) PVDM learns a video distribution in a low-dimensional latent space and thus can be efficiently trained with high-resolution videos under limited resources.
  arXiv  Detail & Related papers  (2023-02-15T14:22:34Z)
• VIDM: Video Implicit Diffusion Models [75.90225524502759]
  Diffusion models have emerged as a powerful generative method for synthesizing high-quality and diverse set of images. We propose a video generation method based on diffusion models, where the effects of motion are modeled in an implicit condition. We improve the quality of the generated videos by proposing multiple strategies such as sampling space truncation, robustness penalty, and positional group normalization.
  arXiv  Detail & Related papers  (2022-12-01T02:58:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.