TLB-VFI: Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation

• URL: http://arxiv.org/abs/2507.04984v1
• Date: Mon, 07 Jul 2025 13:25:32 GMT
• Title: TLB-VFI: Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation
• Authors: Zonglin Lyu, Chen Chen,
• Abstract summary: Video Frame Interpolation (VFI) aims to predict the intermediate frame $I_n$ based on two consecutive neighboring frames.<n>Recent approaches apply diffusion models (both image-based and video-based) in this task and achieve strong performance.<n>We propose Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation (TLB-VFI), an efficient video-based diffusion model.
• Score: 4.261090951843438
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video Frame Interpolation (VFI) aims to predict the intermediate frame $I_n$ (we use n to denote time in videos to avoid notation overload with the timestep $t$ in diffusion models) based on two consecutive neighboring frames $I_0$ and $I_1$. Recent approaches apply diffusion models (both image-based and video-based) in this task and achieve strong performance. However, image-based diffusion models are unable to extract temporal information and are relatively inefficient compared to non-diffusion methods. Video-based diffusion models can extract temporal information, but they are too large in terms of training scale, model size, and inference time. To mitigate the above issues, we propose Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation (TLB-VFI), an efficient video-based diffusion model. By extracting rich temporal information from video inputs through our proposed 3D-wavelet gating and temporal-aware autoencoder, our method achieves 20% improvement in FID on the most challenging datasets over recent SOTA of image-based diffusion models. Meanwhile, due to the existence of rich temporal information, our method achieves strong performance while having 3times fewer parameters. Such a parameter reduction results in 2.3x speed up. By incorporating optical flow guidance, our method requires 9000x less training data and achieves over 20x fewer parameters than video-based diffusion models. Codes and results are available at our project page: https://zonglinl.github.io/tlbvfi_page.

Related papers

• RainFusion: Adaptive Video Generation Acceleration via Multi-Dimensional Visual Redundancy [10.53687668536011]
  RainFusion exploits inherent sparsity nature in visual data to accelerate attention computation while preserving video quality.<n>Our proposed bf RainFusion is a plug-and-play method that can be seamlessly integrated into state-of-the-art 3D-attention video generation models.
  arXiv  Detail & Related papers  (2025-05-27T11:15:02Z)
• SRDiffusion: Accelerate Video Diffusion Inference via Sketching-Rendering Cooperation [26.045123066151838]
  SRDiffusion is a novel framework that leverages collaboration between large and small models to reduce inference cost.<n>Our method is introduced as a new direction to existing acceleration strategies, offering a practical solution for scalable video generation.
  arXiv  Detail & Related papers  (2025-05-25T13:58:52Z)
• Time-adaptive Video Frame Interpolation based on Residual Diffusion [2.5261465733373965]
  We propose a new diffusion-based method for video frame (VFI)<n>In this work, we propose a new diffusion-based method for video frame (VFI)<n>We provide extensive comparisons with respect to state-of-the-art models and show that our model outperforms these models on animation videos.
  arXiv  Detail & Related papers  (2025-04-07T18:15:45Z)
• Token-Efficient Long Video Understanding for Multimodal LLMs [101.70681093383365]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler [53.98558445900626]
  Current image-to-video diffusion models, while powerful in generating videos from a single frame, need adaptation for two-frame conditioned generation.<n>We introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning.<n>Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames.
  arXiv  Detail & Related papers  (2024-10-08T03:01:54Z)
• Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model [31.70050311326183]
  Diffusion models tend to generate videos with less motion than expected. We address this issue from both inference and training aspects. Our methods outperform baselines by producing higher motion scores with lower errors.
  arXiv  Detail & Related papers  (2024-06-22T04:56:16Z)
• ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation [81.90265212988844]
  We propose a training-free video method for generative video models in a plug-and-play manner. We transform a video model into a self-cascaded video diffusion model with the designed hidden state correction modules. Our training-free method is even comparable to trained models supported by huge compute resources and large-scale datasets.
  arXiv  Detail & Related papers  (2024-06-03T00:31:13Z)
• Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition [124.41196697408627]
  We propose content-motion latent diffusion model (CMD), a novel efficient extension of pretrained image diffusion models for video generation. CMD encodes a video as a combination of a content frame (like an image) and a low-dimensional motion latent representation. We generate the content frame by fine-tuning a pretrained image diffusion model, and we generate the motion latent representation by training a new lightweight diffusion model.
  arXiv  Detail & Related papers  (2024-03-21T05:48:48Z)
• Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models [52.93036326078229]
  Off-the-shelf billion-scale datasets for image generation are available, but collecting similar video data of the same scale is still challenging. In this work, we explore finetuning a pretrained image diffusion model with video data as a practical solution for the video synthesis task. Our model, Preserve Your Own Correlation (PYoCo), attains SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks.
  arXiv  Detail & Related papers  (2023-05-17T17:59:16Z)
• NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation [157.07019458623242]
  NUWA-XL is a novel Diffusion over Diffusion architecture for eXtremely Long generation. Our approach adopts a coarse-to-fine'' process, in which the video can be generated in parallel at the same granularity. Experiments show that our model not only generates high-quality long videos with both global and local coherence, but also decreases the average inference time from 7.55min to 26s.
  arXiv  Detail & Related papers  (2023-03-22T07:10:09Z)
• Latent Video Diffusion Models for High-Fidelity Long Video Generation [58.346702410885236]
  We introduce lightweight video diffusion models using a low-dimensional 3D latent space. We also propose hierarchical diffusion in the latent space such that longer videos with more than one thousand frames can be produced. Our framework generates more realistic and longer videos than previous strong baselines.
  arXiv  Detail & Related papers  (2022-11-23T18:58:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.