Mobile Video Diffusion

• URL: http://arxiv.org/abs/2412.07583v1
• Date: Tue, 10 Dec 2024 15:19:10 GMT
• Title: Mobile Video Diffusion
• Abstract summary: Video models have achieved impressive realism and controllability but are limited by computational demands.<n>This paper introduces the first mobile-optimized video diffusion model.<n>Our model, coined as MobileVD, is 523x more efficient (181 vs. 4.34 TFLOs) with a slight quality drop.
• Score: 11.568267488097401
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video diffusion models have achieved impressive realism and controllability but are limited by high computational demands, restricting their use on mobile devices. This paper introduces the first mobile-optimized video diffusion model. Starting from a spatio-temporal UNet from Stable Video Diffusion (SVD), we reduce memory and computational cost by reducing the frame resolution, incorporating multi-scale temporal representations, and introducing two novel pruning schema to reduce the number of channels and temporal blocks. Furthermore, we employ adversarial finetuning to reduce the denoising to a single step. Our model, coined as MobileVD, is 523x more efficient (1817.2 vs. 4.34 TFLOPs) with a slight quality drop (FVD 149 vs. 171), generating latents for a 14x512x256 px clip in 1.7 seconds on a Xiaomi-14 Pro. Our results are available at https://qualcomm-ai-research.github.io/mobile-video-diffusion/

Related papers

• MobileWan: Closing the Quality Gap for Mobile Video Diffusion [53.20920841863923]
  MobileWan is the first 5B-scale video diffusion model deployable on a commercial mobile device.<n>Our system generates 5-second 480x832 videos at 16 FPS in 20 seconds end-to-end latency, achieving a VBench score of 83.79.
  arXiv  Detail & Related papers  (2026-07-07T11:52:46Z)
• Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models [59.36933513963735]
  Diffusion models provide powerful priors for zero-shot video inverse problems.<n>Their real-time deployment is hindered by two inefficiencies: high initial latency caused by holistic video restoration, and low throughput resulting from multiple VAE passes.<n>We propose Autoregressive Video Inverse problem solver (AVIS) to overcome these limitations.
  arXiv  Detail & Related papers  (2026-05-20T02:16:30Z)
• DeCo: Frequency-Decoupled Pixel Diffusion for End-to-End Image Generation [93.6273078684831]
  We propose a frequency-DeCoupled pixel diffusion framework to pursue a more efficient pixel diffusion paradigm.<n>With the intuition to decouple the generation of high and low frequency components, we leverage a lightweight pixel decoder to generate high-frequency details conditioned on semantic guidance.<n>Experiments show that DeCo achieves superior performance among pixel diffusion models, attaining FID of 1.62 (256x256) and 2.22 (512x512) on ImageNet.
  arXiv  Detail & Related papers  (2025-11-24T17:59:06Z)
• Asymmetric VAE for One-Step Video Super-Resolution Acceleration [63.419142632861345]
  We propose FastVSR, which achieves substantial reductions in computational cost by implementing a high compression VAE.<n>FastVSR achieves speedups of 111.9 times compared to multi-step models and 3.92 times compared to existing one-step models.
  arXiv  Detail & Related papers  (2025-09-29T00:36:14Z)
• Turbo-VAED: Fast and Stable Transfer of Video-VAEs to Mobile Devices [36.637983575162075]
  We propose a low-cost solution that efficiently transfers widely used video VAEs to mobile devices.<n>Our method enables real-time 720p video VAE decoding on mobile devices for the first time.<n>Turbo-VAED achieves a 2.9x speedup in FPS and better reconstruction quality on the iPhone 16 Pro.
  arXiv  Detail & Related papers  (2025-08-12T17:59:46Z)
• TLB-VFI: Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation [4.261090951843438]
  Video Frame Interpolation (VFI) aims to predict the intermediate frame $I_n$ based on two consecutive neighboring frames.<n>Recent approaches apply diffusion models (both image-based and video-based) in this task and achieve strong performance.<n>We propose Temporal-Aware Latent Brownian Bridge Diffusion for Video Frame Interpolation (TLB-VFI), an efficient video-based diffusion model.
  arXiv  Detail & Related papers  (2025-07-07T13:25:32Z)
• LLIA -- Enabling Low-Latency Interactive Avatars: Real-Time Audio-Driven Portrait Video Generation with Diffusion Models [17.858801012726445]
  Diffusion-based models have gained wide adoption in the virtual human generation due to their outstanding expressiveness.<n>We present a novel audio-driven portrait video generation framework based on the diffusion model to address these challenges.<n>Our model achieves a maximum of 78 FPS at a resolution of 384x384 and 45 FPS at a resolution of 512x512, with an initial video generation latency of 140 ms and 215 ms, respectively.
  arXiv  Detail & Related papers  (2025-06-06T07:09:07Z)
• LTX-Video: Realtime Video Latent Diffusion [4.7789714048042775]
  LTX-Video is a transformer-based latent diffusion model. It seamlessly integrates the Video-VAE and the denoising transformer. It achieves faster-than-real-time generation, producing 5 seconds of 24 fps video 768 atx512 resolution in just 2 seconds on an Nvidia H100 GPU.
  arXiv  Detail & Related papers  (2024-12-30T19:00:25Z)
• SnapGen-V: Generating a Five-Second Video within Five Seconds on a Mobile Device [61.42406720183769]
  We propose a comprehensive acceleration framework to bring the power of the large-scale video diffusion model to the hands of edge users. Our model, with only 0.6B parameters, can generate a 5-second video on an iPhone 16 PM within 5 seconds.
  arXiv  Detail & Related papers  (2024-12-13T18:59:56Z)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models [52.32078428442281]
  Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies. We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly. Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models.
  arXiv  Detail & Related papers  (2024-12-10T18:59:50Z)
• REDUCIO! Generating 1024$\times$1024 Video within 16 Seconds using Extremely Compressed Motion Latents [110.41795676048835]
  One crucial obstacle for large-scale applications is the expensive training and inference cost. In this paper, we argue that videos contain much more redundant information than images, thus can be encoded by very few motion latents. We train Reducio-DiT in around 3.2K training hours in total and generate a 16-frame 1024*1024 video clip within 15.5 seconds on a single A100 GPU.
  arXiv  Detail & Related papers  (2024-11-20T18:59:52Z)
• Adaptive Caching for Faster Video Generation with Diffusion Transformers [52.73348147077075]
  Diffusion Transformers (DiTs) rely on larger models and heavier attention mechanisms, resulting in slower inference speeds. We introduce a training-free method to accelerate video DiTs, termed Adaptive Caching (AdaCache) We also introduce a Motion Regularization (MoReg) scheme to utilize video information within AdaCache, controlling the compute allocation based on motion content.
  arXiv  Detail & Related papers  (2024-11-04T18:59:44Z)
• Progressive Autoregressive Video Diffusion Models [24.97019070991881]
  We show that existing models can be naturally extended to autoregressive video diffusion models without changing the architectures. We present state-of-the-art results on long video generation at 1 minute (1440 frames at 24 FPS)
  arXiv  Detail & Related papers  (2024-10-10T17:36:15Z)
• Hierarchical Patch Diffusion Models for High-Resolution Video Generation [50.42746357450949]
  We develop deep context fusion, which propagates context information from low-scale to high-scale patches in a hierarchical manner. We also propose adaptive computation, which allocates more network capacity and computation towards coarse image details. The resulting model sets a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in class-conditional video generation.
  arXiv  Detail & Related papers  (2024-06-12T01:12:53Z)
• Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition [124.41196697408627]
  We propose content-motion latent diffusion model (CMD), a novel efficient extension of pretrained image diffusion models for video generation. CMD encodes a video as a combination of a content frame (like an image) and a low-dimensional motion latent representation. We generate the content frame by fine-tuning a pretrained image diffusion model, and we generate the motion latent representation by training a new lightweight diffusion model.
  arXiv  Detail & Related papers  (2024-03-21T05:48:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.