ViBe: Ultra-High-Resolution Video Synthesis Born from Pure Images

• URL: http://arxiv.org/abs/2603.23326v1
• Date: Tue, 24 Mar 2026 15:27:22 GMT
• Title: ViBe: Ultra-High-Resolution Video Synthesis Born from Pure Images
• Abstract summary: Transformer-based video diffusion models rely on 3D attention over spatial and temporal tokens.<n>We propose a pure image adaptation framework that upgrades a video Diffusion Transformer pre-trained at its native scale to synthesize higher-resolution videos.
• Score: 30.646542711556787
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based video diffusion models rely on 3D attention over spatial and temporal tokens, which incurs quadratic time and memory complexity and makes end-to-end training for ultra-high-resolution videos prohibitively expensive. To overcome this bottleneck, we propose a pure image adaptation framework that upgrades a video Diffusion Transformer pre-trained at its native scale to synthesize higher-resolution videos. Unfortunately, naively fine-tuning with high-resolution images alone often introduces noticeable noise due to the image-video modality gap. To address this, we decouple the learning objective to separately handle modality alignment and spatial extrapolation. At the core of our approach is Relay LoRA, a two-stage adaptation strategy. In the first stage, the video diffusion model is adapted to the image domain using low-resolution images to bridge the modality gap. In the second stage, the model is further adapted with high-resolution images to acquire spatial extrapolation capability. During inference, only the high-resolution adaptation is retained to preserve the video generation modality while enabling high-resolution video synthesis. To enhance fine-grained detail synthesis, we further propose a High-Frequency-Awareness-Training-Objective, which explicitly encourages the model to recover high-frequency components from degraded latent representations via a dedicated reconstruction loss. Extensive experiments demonstrate that our method produces ultra-high-resolution videos with rich visual details without requiring any video training data, even outperforming previous state-of-the-art models trained on high-resolution videos by 0.8 on the VBench benchmark. Code will be available at https://github.com/WillWu111/ViBe.

Related papers

• HiReFF: High-Resolution Feedforward Human Reconstruction from Uncalibrated Sparse-View Video [58.85548854304172]
  HiReFF is a feed-forward method for 2K-resolution 360 human video reconstruction from uncalibrated sparse-view videos.<n>Our framework decomposes the problem into two key tasks: 3D Gaussian reconstruction from sparse-view videos and computationally efficient high-resolution synthesis.<n> Experiments demonstrate that HiReFF significantly outperforms existing methods in high-resolution streaming video reconstruction.
  arXiv  Detail & Related papers  (2026-06-28T11:01:36Z)
• CineScale: Free Lunch in High-Resolution Cinematic Visual Generation [42.81729840016782]
  We propose CineScale, a novel inference paradigm to enable higher-resolution visual generation.<n>Our approach enables 8k image generation without any fine-tuning, and achieves 4k video generation with only minimal LoRA fine-tuning.
  arXiv  Detail & Related papers  (2025-08-21T17:59:57Z)
• Super-Resolution Generative Adversarial Networks based Video Enhancement [0.40964539027092906]
  This work introduces an enhanced approach to video super-resolution by extending ordinary Single-Image-SIS (SRGAN) structure to handle-versarial data.<n>A modified framework that incorporates 3D Non-Local Blocks is developed, which is enabling the model to capture relationships across both spatial and temporal dimensions.<n>Results show improved temporal coherence, sharper textures, and fewer visual artifacts compared to traditional single-image methods.
  arXiv  Detail & Related papers  (2025-05-14T20:16:51Z)
• STAR: Spatial-Temporal Augmentation with Text-to-Video Models for Real-World Video Super-Resolution [42.859188375578604]
  Image diffusion models have been adapted for real-world video superresolution to tackle over-smoothing issues in GAN-based methods.<n>These models struggle to maintain temporal consistency, as they are trained on static images.<n>We introduce a novel approach that leverages T2V models for real-world video super-resolution, achieving realistic spatial details and robust temporal consistency.
  arXiv  Detail & Related papers  (2025-01-06T12:36:21Z)
• VISION-XL: High Definition Video Inverse Problem Solver using Latent Image Diffusion Models [58.464465016269614]
  We propose a novel framework for solving high-definition video inverse problems using latent image diffusion models.<n>Our approach delivers HD-resolution reconstructions in under 6 seconds per frame on a single NVIDIA 4090 GPU.
  arXiv  Detail & Related papers  (2024-11-29T08:10:49Z)
• Hierarchical Patch Diffusion Models for High-Resolution Video Generation [50.42746357450949]
  We develop deep context fusion, which propagates context information from low-scale to high-scale patches in a hierarchical manner. We also propose adaptive computation, which allocates more network capacity and computation towards coarse image details. The resulting model sets a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in class-conditional video generation.
  arXiv  Detail & Related papers  (2024-06-12T01:12:53Z)
• Learning Spatial Adaptation and Temporal Coherence in Diffusion Models for Video Super-Resolution [151.1255837803585]
  We propose a novel approach, pursuing Spatial Adaptation and Temporal Coherence (SATeCo) for video super-resolution. SATeCo pivots on learning spatial-temporal guidance from low-resolution videos to calibrate both latent-space high-resolution video denoising and pixel-space video reconstruction. Experiments conducted on the REDS4 and Vid4 datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2024-03-25T17:59:26Z)
• VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation [73.54366331493007]
  VideoGen is a text-to-video generation approach, which can generate a high-definition video with high frame fidelity and strong temporal consistency. We leverage an off-the-shelf text-to-image generation model, e.g., Stable Diffusion, to generate an image with high content quality from the text prompt.
  arXiv  Detail & Related papers  (2023-09-01T11:14:43Z)
• Towards Interpretable Video Super-Resolution via Alternating Optimization [115.85296325037565]
  We study a practical space-time video super-resolution (STVSR) problem which aims at generating a high-framerate high-resolution sharp video from a low-framerate blurry video. We propose an interpretable STVSR framework by leveraging both model-based and learning-based methods.
  arXiv  Detail & Related papers  (2022-07-21T21:34:05Z)
• High Fidelity Interactive Video Segmentation Using Tensor Decomposition Boundary Loss Convolutional Tessellations and Context Aware Skip Connections [0.0]
  We provide a high fidelity deep learning algorithm (HyperSeg) for interactive video segmentation tasks. Our model crucially processes and renders all image features in high resolution, without utilizing downsampling or pooling procedures. Our work can be used across a broad range of application domains, including VFX pipelines and medical imaging disciplines.
  arXiv  Detail & Related papers  (2020-11-23T18:21:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.