Augmented Deep Contexts for Spatially Embedded Video Coding

• URL: http://arxiv.org/abs/2505.05309v1
• Date: Thu, 08 May 2025 14:57:52 GMT
• Title: Augmented Deep Contexts for Spatially Embedded Video Coding
• Abstract summary: Most Neural Video Codecs (NVCs) only employ temporal references to generate temporal-only contexts and latent prior.<n>We propose a Spatially Embedded Video Codec (SEVC) in which the low-resolution video is compressed for spatial references.<n>Our SEVC effectively alleviates the limitations in handling large motions or emerging objects, and also reduces 11.9% more than the previous state-of-the-art NVC.
• Score: 8.213635577747638
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most Neural Video Codecs (NVCs) only employ temporal references to generate temporal-only contexts and latent prior. These temporal-only NVCs fail to handle large motions or emerging objects due to limited contexts and misaligned latent prior. To relieve the limitations, we propose a Spatially Embedded Video Codec (SEVC), in which the low-resolution video is compressed for spatial references. Firstly, our SEVC leverages both spatial and temporal references to generate augmented motion vectors and hybrid spatial-temporal contexts. Secondly, to address the misalignment issue in latent prior and enrich the prior information, we introduce a spatial-guided latent prior augmented by multiple temporal latent representations. At last, we design a joint spatial-temporal optimization to learn quality-adaptive bit allocation for spatial references, further boosting rate-distortion performance. Experimental results show that our SEVC effectively alleviates the limitations in handling large motions or emerging objects, and also reduces 11.9% more bitrate than the previous state-of-the-art NVC while providing an additional low-resolution bitstream. Our code and model are available at https://github.com/EsakaK/SEVC.

Related papers

• Locate Anything in Videos: Rethinking Efficient Generative Spatio-Temporal Video Grounding [80.10744910189997]
  multimodal large language models typically serialize trajectories autoregressively.<n>We introduce Parallel Tube Decoding (PTD), a generative formulation that decomposes grounding into a temporal block.<n>On VidSTG, PTD Tube reduces Completion by 79x and increases spatial decoding by 92x over standard autoregressive decoding.
  arXiv  Detail & Related papers  (2026-08-28T11:05:30Z)
• Dynamic Video Generation: Shaping Video Generation Across Time and Space [9.269520155517094]
  We propose DVG, a framework that automatically selects content-aware acceleration strategies without manual tuning or retraining.<n> DVG achieves near-lossless acceleration across models and tasks, reaching up to 7 times speedup on HunyuanVideo and HunyuanVideo-1.51.5, and 18 times when combined with distillation.
  arXiv  Detail & Related papers  (2026-05-20T11:24:02Z)
• Generative Video Compression with One-Dimensional Latent Representation [59.29493435530304]
  We introduce Generative Video Compression with One-Dimensional (1D) Latent Representation (GVC1D)<n>GVC1D encodes the video data into extreme compact 1D latent tokens conditioned on both short- and long-term contexts.<n> Experimental results indicate that GVC1D attains superior compression efficiency, surpassing previous video compression methods.
  arXiv  Detail & Related papers  (2026-03-16T14:02:17Z)
• TIMERIPPLE: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space [15.535854202219072]
  We take a principled approach to accelerate self-attention in vDiTs by leveraging the reuse-temporal correlations in the latent space.<n>We show that the attention patterns within vDiT are primarily due to the dominant spatial and temporal correlations at the token channel level.<n>We propose a lightweight and adaptive strategy that approximates attention computations by reusing partial attention scores spatially or temporally correlated tokens along individual channels.
  arXiv  Detail & Related papers  (2025-11-15T05:07:31Z)
• Generative Latent Video Compression [26.99743586846841]
  We present Generative Latent Video Compression (GLVC), an effective framework for perceptual video compression.<n>GLVC employs a pretrained continuous tokenizer to project video frames into a perceptually aligned latent space.<n>We show GLVC achieves state-of-the-art performance in terms of DISTS and LPIPS metrics.
  arXiv  Detail & Related papers  (2025-10-11T03:28:49Z)
• Continuous Space-Time Video Super-Resolution with 3D Fourier Fields [62.270473766381976]
  We introduce a novel formulation for continuous space-time video super-resolution.<n>We show that our modeling joint substantially improves both spatial and temporal super-resolution.
  arXiv  Detail & Related papers  (2025-09-30T14:34:02Z)
• Lightning Fast Caching-based Parallel Denoising Prediction for Accelerating Talking Head Generation [50.04968365065964]
  Diffusion-based talking head models generate high-quality, photorealistic videos but suffer from slow inference.<n>We introduce Lightning-fast Caching-based Parallel denoising prediction (LightningCP)<n>We also propose Decoupled Foreground Attention (DFA) to further accelerate attention computations.
  arXiv  Detail & Related papers  (2025-08-25T02:58:39Z)
• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)
• Neural Video Compression with Context Modulation [9.875413481663742]
  In this paper, we address the limitation by modulating the temporal context with the reference frame in two steps.<n>We achieve on average 22.7% reduction over the advanced traditional video H.266/VVC, and offer an average 10.1% saving over the previous state-of-the-art NVC DCVC-FM.
  arXiv  Detail & Related papers  (2025-05-20T15:57:09Z)
• BiECVC: Gated Diversification of Bidirectional Contexts for Learned Video Compression [12.60355288519781]
  We propose BiECVC, a learned bidirectional video compression (BVC) framework that incorporates diversified local and non-local context modeling.<n>BiECVC achieves state-of-the-art performance, reducing the bit-rate by 13.4% and 15.7% compared to VTM 13.2 under the Random Access (RA) configuration.<n>To our knowledge BiECVC is the first learned video to surpass VTM 13.2 across all standard test datasets.
  arXiv  Detail & Related papers  (2025-05-14T06:55:37Z)
• Exploiting Temporal State Space Sharing for Video Semantic Segmentation [53.8810901249897]
  Video semantic segmentation (VSS) plays a vital role in understanding the temporal evolution of scenes.<n>Traditional methods often segment videos frame-by-frame or in a short temporal window, leading to limited temporal context, redundant computations, and heavy memory requirements.<n>We introduce a Temporal Video State Space Sharing architecture to leverage Mamba state space models for temporal feature sharing.<n>Our model features a selective gating mechanism that efficiently propagates relevant information across video frames, eliminating the need for a memory-heavy feature pool.
  arXiv  Detail & Related papers  (2025-03-26T01:47:42Z)
• Towards Practical Real-Time Neural Video Compression [60.390180067626396]
  We introduce a practical real-time neural video (NVC) designed to deliver high compression ratio, low latency and broad versatility.<n>Experiments show our proposed DCVC-RT achieves an impressive average encoding/desampling speed 125.2/112.8 (frames per second) for 1080p video, while saving an average of 21% in fps compared to H.266/VTM.
  arXiv  Detail & Related papers  (2025-02-28T06:32:23Z)
• BF-STVSR: B-Splines and Fourier-Best Friends for High Fidelity Spatial-Temporal Video Super-Resolution [14.082598088990352]
  We propose BF-STVSR, a C-STVSR framework with two key modules tailored to better represent spatial and temporal characteristics of video.<n>Our approach achieves state-of-the-art in various metrics, including PSNR and SSIM, showing enhanced spatial details and natural temporal consistency.
  arXiv  Detail & Related papers  (2025-01-19T13:29:41Z)
• Large Motion Video Autoencoding with Cross-modal Video VAE [52.13379965800485]
  Video Variational Autoencoder (VAE) is essential for reducing video redundancy and facilitating efficient video generation.<n>Existing Video VAEs have begun to address temporal compression; however, they often suffer from inadequate reconstruction performance.<n>We present a novel and powerful video autoencoder capable of high-fidelity video encoding.
  arXiv  Detail & Related papers  (2024-12-23T18:58:24Z)
• Spatio-Temporal Distortion Aware Omnidirectional Video Super-Resolution [25.615935776826596]
  Omnidirectional (ODVs) provide an immersive visual experience by capturing the 360deg scene.<n>ODVs often suffer from low resolution due to their wide field of view and limitations in capturing devices and transmission bandwidth.<n>We propose a Spatio-Temporal Aware Distortion Network (STDAN) to alleviate spatial projection distortions and temporal flickering of ODVs.
  arXiv  Detail & Related papers  (2024-10-15T11:17:19Z)
• Spatial Decomposition and Temporal Fusion based Inter Prediction for Learned Video Compression [59.632286735304156]
  We propose a spatial decomposition and temporal fusion based inter prediction for learned video compression. With the SDD-based motion model and long short-term temporal fusion, our proposed learned video can obtain more accurate inter prediction contexts.
  arXiv  Detail & Related papers  (2024-01-29T03:30:21Z)
• Orthogonal Temporal Interpolation for Zero-Shot Video Recognition [45.53856045374685]
  Zero-shot video recognition (ZSVR) is a task that aims to recognize video categories that have not been seen during the model training process. Recent vision-language models (VLMs) pre-trained on large-scale image-text pairs have demonstrated impressive transferability for ZSVR.
  arXiv  Detail & Related papers  (2023-08-14T02:26:49Z)
• Neural Video Compression with Diverse Contexts [25.96187914295921]
  This paper proposes increasing the context diversity in both temporal and spatial dimensions. Experiments show that our codes obtains 23.5% saving over previous SOTA NVC.
  arXiv  Detail & Related papers  (2023-02-28T08:35:50Z)
• VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution [75.79379734567604]
  We show that Video Implicit Neural Representation (VideoINR) can be decoded to videos of arbitrary spatial resolution and frame rate. We show that VideoINR achieves competitive performances with state-of-the-art STVSR methods on common up-sampling scales.
  arXiv  Detail & Related papers  (2022-06-09T17:45:49Z)
• Decomposition, Compression, and Synthesis (DCS)-based Video Coding: A Neural Exploration via Resolution-Adaptive Learning [30.54722074562783]
  We decompose the input video into respective spatial texture frames (STF) at its native spatial resolution. Then, we compress them together using any popular video coder. Finally, we synthesize decoded STFs and TMFs for high-quality video reconstruction at the same resolution as its native input.
  arXiv  Detail & Related papers  (2020-12-01T17:23:53Z)
• Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model [45.46660511313426]
  We propose an end-to-end deep neural video coding framework (NVC) It uses variational autoencoders (VAEs) with joint spatial and temporal prior aggregation (PA) to exploit the correlations in intra-frame pixels, inter-frame motions and inter-frame compensation residuals. NVC is evaluated for the low-delay causal settings and compared with H.265/HEVC, H.264/AVC and the other learnt video compression methods.
  arXiv  Detail & Related papers  (2020-07-09T06:15:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.