Augmented Deep Contexts for Spatially Embedded Video Coding

• URL: http://arxiv.org/abs/2505.05309v1
• Date: Thu, 08 May 2025 14:57:52 GMT
• Title: Augmented Deep Contexts for Spatially Embedded Video Coding
• Authors: Yifan Bian, Chuanbo Tang, Li Li, Dong Liu,
• Abstract summary: Most Neural Video Codecs (NVCs) only employ temporal references to generate temporal-only contexts and latent prior.<n>We propose a Spatially Embedded Video Codec (SEVC) in which the low-resolution video is compressed for spatial references.<n>Our SEVC effectively alleviates the limitations in handling large motions or emerging objects, and also reduces 11.9% more than the previous state-of-the-art NVC.
• Score: 8.213635577747638
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most Neural Video Codecs (NVCs) only employ temporal references to generate temporal-only contexts and latent prior. These temporal-only NVCs fail to handle large motions or emerging objects due to limited contexts and misaligned latent prior. To relieve the limitations, we propose a Spatially Embedded Video Codec (SEVC), in which the low-resolution video is compressed for spatial references. Firstly, our SEVC leverages both spatial and temporal references to generate augmented motion vectors and hybrid spatial-temporal contexts. Secondly, to address the misalignment issue in latent prior and enrich the prior information, we introduce a spatial-guided latent prior augmented by multiple temporal latent representations. At last, we design a joint spatial-temporal optimization to learn quality-adaptive bit allocation for spatial references, further boosting rate-distortion performance. Experimental results show that our SEVC effectively alleviates the limitations in handling large motions or emerging objects, and also reduces 11.9% more bitrate than the previous state-of-the-art NVC while providing an additional low-resolution bitstream. Our code and model are available at https://github.com/EsakaK/SEVC.

Related papers

• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)
• Neural Video Compression with Context Modulation [9.875413481663742]
  In this paper, we address the limitation by modulating the temporal context with the reference frame in two steps.<n>We achieve on average 22.7% reduction over the advanced traditional video H.266/VVC, and offer an average 10.1% saving over the previous state-of-the-art NVC DCVC-FM.
  arXiv  Detail & Related papers  (2025-05-20T15:57:09Z)
• BiECVC: Gated Diversification of Bidirectional Contexts for Learned Video Compression [12.60355288519781]
  We propose BiECVC, a learned bidirectional video compression (BVC) framework that incorporates diversified local and non-local context modeling.<n>BiECVC achieves state-of-the-art performance, reducing the bit-rate by 13.4% and 15.7% compared to VTM 13.2 under the Random Access (RA) configuration.<n>To our knowledge BiECVC is the first learned video to surpass VTM 13.2 across all standard test datasets.
  arXiv  Detail & Related papers  (2025-05-14T06:55:37Z)
• Exploiting Temporal State Space Sharing for Video Semantic Segmentation [53.8810901249897]
  Video semantic segmentation (VSS) plays a vital role in understanding the temporal evolution of scenes.<n>Traditional methods often segment videos frame-by-frame or in a short temporal window, leading to limited temporal context, redundant computations, and heavy memory requirements.<n>We introduce a Temporal Video State Space Sharing architecture to leverage Mamba state space models for temporal feature sharing.<n>Our model features a selective gating mechanism that efficiently propagates relevant information across video frames, eliminating the need for a memory-heavy feature pool.
  arXiv  Detail & Related papers  (2025-03-26T01:47:42Z)
• Towards Practical Real-Time Neural Video Compression [60.390180067626396]
  We introduce a practical real-time neural video (NVC) designed to deliver high compression ratio, low latency and broad versatility.<n>Experiments show our proposed DCVC-RT achieves an impressive average encoding/desampling speed 125.2/112.8 (frames per second) for 1080p video, while saving an average of 21% in fps compared to H.266/VTM.
  arXiv  Detail & Related papers  (2025-02-28T06:32:23Z)
• BF-STVSR: B-Splines and Fourier-Best Friends for High Fidelity Spatial-Temporal Video Super-Resolution [14.082598088990352]
  We propose BF-STVSR, a C-STVSR framework with two key modules tailored to better represent spatial and temporal characteristics of video.<n>Our approach achieves state-of-the-art in various metrics, including PSNR and SSIM, showing enhanced spatial details and natural temporal consistency.
  arXiv  Detail & Related papers  (2025-01-19T13:29:41Z)
• Large Motion Video Autoencoding with Cross-modal Video VAE [52.13379965800485]
  Video Variational Autoencoder (VAE) is essential for reducing video redundancy and facilitating efficient video generation.<n>Existing Video VAEs have begun to address temporal compression; however, they often suffer from inadequate reconstruction performance.<n>We present a novel and powerful video autoencoder capable of high-fidelity video encoding.
  arXiv  Detail & Related papers  (2024-12-23T18:58:24Z)
• Spatio-Temporal Distortion Aware Omnidirectional Video Super-Resolution [25.615935776826596]
  Omnidirectional (ODVs) provide an immersive visual experience by capturing the 360deg scene.<n>ODVs often suffer from low resolution due to their wide field of view and limitations in capturing devices and transmission bandwidth.<n>We propose a Spatio-Temporal Aware Distortion Network (STDAN) to alleviate spatial projection distortions and temporal flickering of ODVs.
  arXiv  Detail & Related papers  (2024-10-15T11:17:19Z)
• Spatial Decomposition and Temporal Fusion based Inter Prediction for Learned Video Compression [59.632286735304156]
  We propose a spatial decomposition and temporal fusion based inter prediction for learned video compression. With the SDD-based motion model and long short-term temporal fusion, our proposed learned video can obtain more accurate inter prediction contexts.
  arXiv  Detail & Related papers  (2024-01-29T03:30:21Z)
• Orthogonal Temporal Interpolation for Zero-Shot Video Recognition [45.53856045374685]
  Zero-shot video recognition (ZSVR) is a task that aims to recognize video categories that have not been seen during the model training process. Recent vision-language models (VLMs) pre-trained on large-scale image-text pairs have demonstrated impressive transferability for ZSVR.
  arXiv  Detail & Related papers  (2023-08-14T02:26:49Z)
• Neural Video Compression with Diverse Contexts [25.96187914295921]
  This paper proposes increasing the context diversity in both temporal and spatial dimensions. Experiments show that our codes obtains 23.5% saving over previous SOTA NVC.
  arXiv  Detail & Related papers  (2023-02-28T08:35:50Z)
• VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution [75.79379734567604]
  We show that Video Implicit Neural Representation (VideoINR) can be decoded to videos of arbitrary spatial resolution and frame rate. We show that VideoINR achieves competitive performances with state-of-the-art STVSR methods on common up-sampling scales.
  arXiv  Detail & Related papers  (2022-06-09T17:45:49Z)
• Decomposition, Compression, and Synthesis (DCS)-based Video Coding: A Neural Exploration via Resolution-Adaptive Learning [30.54722074562783]
  We decompose the input video into respective spatial texture frames (STF) at its native spatial resolution. Then, we compress them together using any popular video coder. Finally, we synthesize decoded STFs and TMFs for high-quality video reconstruction at the same resolution as its native input.
  arXiv  Detail & Related papers  (2020-12-01T17:23:53Z)
• Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model [45.46660511313426]
  We propose an end-to-end deep neural video coding framework (NVC) It uses variational autoencoders (VAEs) with joint spatial and temporal prior aggregation (PA) to exploit the correlations in intra-frame pixels, inter-frame motions and inter-frame compensation residuals. NVC is evaluated for the low-delay causal settings and compared with H.265/HEVC, H.264/AVC and the other learnt video compression methods.
  arXiv  Detail & Related papers  (2020-07-09T06:15:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.