DiffVC-RT: Towards Practical Real-Time Diffusion-based Perceptual Neural Video Compression

• URL: http://arxiv.org/abs/2601.20564v1
• Date: Wed, 28 Jan 2026 12:59:25 GMT
• Title: DiffVC-RT: Towards Practical Real-Time Diffusion-based Perceptual Neural Video Compression
• Abstract summary: We present DiffVC-RT, the first framework designed to achieve real-time diffusion-based Neural Video Compression (NVC)<n>We show that DiffVC-RT achieves 80.1% perceptual savings in terms of LPIPS over VTM-17.0 on HEVC dataset with real-time encoding and decoding speeds of 206 / 30 fps for 720p videos on an NVIDIA H800 GPU.
• Score: 38.495966630021556
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The practical deployment of diffusion-based Neural Video Compression (NVC) faces critical challenges, including severe information loss, prohibitive inference latency, and poor temporal consistency. To bridge this gap, we propose DiffVC-RT, the first framework designed to achieve real-time diffusion-based perceptual NVC. First, we introduce an Efficient and Informative Model Architecture. Through strategic module replacements and pruning, this architecture significantly reduces computational complexity while mitigating structural information loss. Second, to address generative flickering artifacts, we propose Explicit and Implicit Consistency Modeling. We enhance temporal consistency by explicitly incorporating a zero-cost Online Temporal Shift Module within the U-Net, complemented by hybrid implicit consistency constraints. Finally, we present an Asynchronous and Parallel Decoding Pipeline incorporating Mixed Half Precision, which enables asynchronous latent decoding and parallel frame reconstruction via a Batch-dimension Temporal Shift design. Experiments show that DiffVC-RT achieves 80.1% bitrate savings in terms of LPIPS over VTM-17.0 on HEVC dataset with real-time encoding and decoding speeds of 206 / 30 fps for 720p videos on an NVIDIA H800 GPU, marking a significant milestone in diffusion-based video compression.

Related papers

• VoRTeC: Taming Foundation Flow for One-step Real time Video Compression [38.79321243531853]
  $mathttVoRTeC$ is a Video Compression framework built upon a foundational flow model (Wan2.1)<n>Our method reduces bit consumption by 58% compared to prior diffusion-based approaches.<n>$mathttVoRTeC$ achieves a decoding speed of 13 FPS at 720p and 32 FPS at 480p.
  arXiv  Detail & Related papers  (2026-09-02T08:39:23Z)
• Ultra-Fast Neural Video Compression [29.065189275652944]
  This paper introduces a chunk-based coding framework designed to significantly improve the rate-distortion-complexity trade-off.<n>Instead of processing frames sequentially, our approach encodes a chunk of multiple frames into a single compact latent representation and decodes them simultaneously.<n>Building on these innovations, we present DCVC-UF, a new NVC that sets a new SOTA in performance.
  arXiv  Detail & Related papers  (2026-06-03T03:38:05Z)
• MOSS-Video-Preview: Toward Real-Time Video Understanding via Cross-Attention [53.80990122465395]
  We show how a cross-attention backbone is better suited to real-time vision-language fusion than the prevailing decoder-only design.<n>Our model trails the strong Qwen2.5-VL-7B baseline overall.<n>On a single H200 with 256 frames per video, achieves it about a 5x speedup in time to first token.
  arXiv  Detail & Related papers  (2026-06-01T09:07:15Z)
• SANA-Streaming: Real-time Streaming Video Editing with Hybrid Diffusion Transformer [36.01290871589722]
  Real-time streaming video-to-video editing is critical for interactive applications such as live broadcasting and gaming.<n>We present SANA-Streaming, a system-algorithm co-designed framework for high-resolution, real-time streaming video editing.
  arXiv  Detail & Related papers  (2026-05-28T17:59:17Z)
• Efficient Video Diffusion with Sparse Information Transmission for Video Compression [24.272279529510772]
  Video compression aims to maximize reconstruction quality with minimal distortions.<n>At ultra-lows, traditional end-to-end compression models tend to produce blurry images of poor quality.<n>We propose Diff-SIT to establish a new state-of-the-art in perceptual quality and temporal consistency.
  arXiv  Detail & Related papers  (2026-03-19T05:15:29Z)
• Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformers [95.68243351895107]
  We propose a holistic, video-centric paradigm named textbfLocal textbfDiffusion textbfForcing for textbfVideo textbfFrame textbfInterpolation (LDF-VFI)<n>Our framework is built upon an auto-regressive diffusion transformer that models the entire video sequence to ensure long-range temporal coherence.<n>LDF-VFI achieves state-of-the-art performance on challenging long-sequence benchmarks, demonstrating superior per
  arXiv  Detail & Related papers  (2026-01-21T12:58:52Z)
• VideoCompressa: Data-Efficient Video Understanding via Joint Temporal Compression and Spatial Reconstruction [55.66673587952058]
  Video understanding models are increasingly limited by the prohibitive storage and computational costs of large-scale datasets.<n>VideoCompressa is a novel framework for video data synthesis that reframes the problem as dynamic latent compression.
  arXiv  Detail & Related papers  (2025-11-24T07:07:58Z)
• Real-Time Neural Video Compression with Unified Intra and Inter Coding [8.998142257336674]
  We present an NVC framework with unified intra and inter coding, where every frame is processed by a single model.<n>We propose a simultaneous two-frame compression design to exploit interframe redundancy not only forwardly but also backwardly.<n>Our scheme outperforms DCVC-RT by an average of 12.1% BD-rate reduction, delivers more stable and quality per frame, and retains real-time encoding/decoding performances.
  arXiv  Detail & Related papers  (2025-10-16T08:31:44Z)
• DiffVC-OSD: One-Step Diffusion-based Perceptual Neural Video Compression Framework [45.134271969594614]
  We first propose DiffVC-OSD, a One-Step Diffusion-based Perceptual Neural Video Compression framework.<n>We employ an End-to-End Finetuning strategy to improve overall compression performance.
  arXiv  Detail & Related papers  (2025-08-11T06:59:23Z)
• FCA2: Frame Compression-Aware Autoencoder for Modular and Fast Compressed Video Super-Resolution [68.77813885751308]
  State-of-the-art (SOTA) compressed video super-resolution (CVSR) models face persistent challenges, including prolonged inference time, complex training pipelines, and reliance on auxiliary information.<n>We propose an efficient and scalable solution inspired by the structural and statistical similarities between hyperspectral images (HSI) and video data.<n>Our approach introduces a compression-driven dimensionality reduction strategy that reduces computational complexity, accelerates inference, and enhances the extraction of temporal information across frames.
  arXiv  Detail & Related papers  (2025-06-13T07:59:52Z)
• Rethinking Video Tokenization: A Conditioned Diffusion-based Approach [58.164354605550194]
  New tokenizer, Diffusion Conditioned-based Gene Tokenizer, replaces GAN-based decoder with conditional diffusion model.<n>We trained using only a basic MSE diffusion loss for reconstruction, along with KL term and LPIPS perceptual loss from scratch.<n>Even a scaled-down version of CDT (3$times inference speedup) still performs comparably with top baselines.
  arXiv  Detail & Related papers  (2025-03-05T17:59:19Z)
• Fast-Vid2Vid: Spatial-Temporal Compression for Video-to-Video Synthesis [40.249030338644225]
  Video-to-Video synthesis (Vid2Vid) has achieved remarkable results in generating a photo-realistic video from a sequence of semantic maps. Fast-Vid2Vid achieves around real-time performance as 20 FPS and saves around 8x computational cost on a single V100 GPU.
  arXiv  Detail & Related papers  (2022-07-11T17:57:57Z)
• Learned Video Compression via Heterogeneous Deformable Compensation Network [78.72508633457392]
  We propose a learned video compression framework via heterogeneous deformable compensation strategy (HDCVC) to tackle the problems of unstable compression performance. More specifically, the proposed algorithm extracts features from the two adjacent frames to estimate content-Neighborhood heterogeneous deformable (HetDeform) kernel offsets. Experimental results indicate that HDCVC achieves superior performance than the recent state-of-the-art learned video compression approaches.
  arXiv  Detail & Related papers  (2022-07-11T02:31:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.