Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models

• URL: http://arxiv.org/abs/2605.20624v1
• Date: Wed, 20 May 2026 02:16:30 GMT
• Title: Accelerating Video Inverse Problem Solvers with Autoregressive Diffusion Models
• Abstract summary: Diffusion models provide powerful priors for zero-shot video inverse problems.<n>Their real-time deployment is hindered by two inefficiencies: high initial latency caused by holistic video restoration, and low throughput resulting from multiple VAE passes.<n>We propose Autoregressive Video Inverse problem solver (AVIS) to overcome these limitations.
• Score: 59.36933513963735
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models provide powerful priors for zero-shot video inverse problems, but their real-time deployment is hindered by two inefficiencies: high initial latency caused by holistic video restoration, and low throughput resulting from multiple VAE passes to enforce measurement consistency in pixel space. To overcome these limitations, we propose Autoregressive Video Inverse problem Solver (AVIS). The AVIS framework leverages autoregressive video diffusion models to restore videos in a streaming manner, naturally eliminating latency bottlenecks. Specifically, AVIS initializes reverse diffusion with a measurement-consistent estimate, reducing the required sampling steps. Compared to leading non-autoregressive solvers, AVIS drastically reduces initial latency from 114s to 4s and increases throughput from 0.71 to 1.18 FPS while achieving superior restoration quality. We further introduce a highly accelerated variant, dubbed AVIS Flash, that enforces measurement consistency solely on the first chunk. AVIS Flash substantially boosts throughput to 5.91 FPS on a single RTX 4090 GPU while maintaining competitive performance and achieving a favorable efficiency-performance trade-off, paving the way toward real-time deployment.

Related papers

• MobileWan: Closing the Quality Gap for Mobile Video Diffusion [53.20920841863923]
  MobileWan is the first 5B-scale video diffusion model deployable on a commercial mobile device.<n>Our system generates 5-second 480x832 videos at 16 FPS in 20 seconds end-to-end latency, achieving a VBench score of 83.79.
  arXiv  Detail & Related papers  (2026-07-07T11:52:46Z)
• DSA: Dynamic Step Allocation for Fast Autoregressive Video Generation [24.843472730316503]
  Video diffusion transformers have achieved state-of-the-art visual quality, but their high inference cost remains a major bottleneck for real-time applications.<n>We present DSA, a confidence-guided adaptive framework for AR video diffusion.
  arXiv  Detail & Related papers  (2026-06-03T04:25:11Z)
• DiffVC-RT: Towards Practical Real-Time Diffusion-based Perceptual Neural Video Compression [38.495966630021556]
  We present DiffVC-RT, the first framework designed to achieve real-time diffusion-based Neural Video Compression (NVC)<n>We show that DiffVC-RT achieves 80.1% perceptual savings in terms of LPIPS over VTM-17.0 on HEVC dataset with real-time encoding and decoding speeds of 206 / 30 fps for 720p videos on an NVIDIA H800 GPU.
  arXiv  Detail & Related papers  (2026-01-28T12:59:25Z)
• Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion [10.847237180991948]
  Stream-DiffVSR is a causally conditioned diffusion framework for efficient online VSR.<n>It processes 720p frames in 0.328 seconds on an GTX4090 GPU.<n>It boosts perceptual quality (LPIPS +0.095) while reducing latency by over 130x.
  arXiv  Detail & Related papers  (2025-12-29T18:59:57Z)
• InstantViR: Real-Time Video Inverse Problem Solver with Distilled Diffusion Prior [13.775331675468024]
  We introduce InstantViR, an amortized inference framework for ultra-fast video reconstruction powered by a pre-trained video diffusion prior.<n>We show that InstantViR is compatible with real-time, interactive, editable, streaming scenarios, turning high-quality video restoration into a practical component of modern vision systems.
  arXiv  Detail & Related papers  (2025-11-18T07:40:38Z)
• FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolution [61.284842030283464]
  FlashVSR is the first diffusion-based one-step streaming framework towards real-time VSR.<n>It runs at approximately 17 FPS for 768x1408 videos on a single A100 GPU.<n>It scales reliably to ultra-high resolutions and achieves state-of-the-art performance with up to 12x speedup over prior one-step diffusion VSR models.
  arXiv  Detail & Related papers  (2025-10-14T17:25:54Z)
• Real-Time Motion-Controllable Autoregressive Video Diffusion [79.32730467857535]
  We propose AR-Drag, the first RL-enhanced few-step AR video diffusion model for real-time image-to-video generation with diverse motion control.<n>We first fine-tune a base I2V model to support basic motion control, then further improve it via reinforcement with a trajectory-based reward model.<n>Our design preserves the Markov property through a Self-Rollout learning mechanism and accelerates training by selectively denoising steps.
  arXiv  Detail & Related papers  (2025-10-09T12:17:11Z)
• Asymmetric VAE for One-Step Video Super-Resolution Acceleration [63.419142632861345]
  We propose FastVSR, which achieves substantial reductions in computational cost by implementing a high compression VAE.<n>FastVSR achieves speedups of 111.9 times compared to multi-step models and 3.92 times compared to existing one-step models.
  arXiv  Detail & Related papers  (2025-09-29T00:36:14Z)
• BLADE: Block-Sparse Attention Meets Step Distillation for Efficient Video Generation [27.57431718095974]
  We present BLADE, a data-free joint training framework for video inference.<n>We show remarkable efficiency gains across different scales.<n>On models such as CogVideoX-5B with short video sequence lengths, our framework delivers a robust 8.89x speedup.
  arXiv  Detail & Related papers  (2025-08-14T15:58:59Z)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models [48.35054927704544]
  Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies.<n>We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly.<n>Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models.
  arXiv  Detail & Related papers  (2024-12-10T18:59:50Z)
• StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation [52.56469577812338]
  We introduce StreamDiffusion, a real-time diffusion pipeline for interactive image generation.<n>Existing diffusion models are adept at creating images from text or image prompts, yet they often fall short in real-time interaction.<n>We present a novel approach that transforms the original sequential denoising into the denoising process.
  arXiv  Detail & Related papers  (2023-12-19T18:18:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.