Robust Multiple Description Neural Video Codec with Masked Transformer for Dynamic and Noisy Networks

• URL: http://arxiv.org/abs/2412.07922v1
• Date: Tue, 10 Dec 2024 21:08:05 GMT
• Title: Robust Multiple Description Neural Video Codec with Masked Transformer for Dynamic and Noisy Networks
• Authors: Xinyue Hu, Wei Ye, Jiaxiang Tang, Eman Ramadan, Zhi-Li Zhang,
• Abstract summary: Multiple Description Coding (MDC) is a promising error-resilient source coding method. Conventional MDC video codecs suffer from cumbersome architectures, poor scalability, limited loss resilience, and lower compression efficiency. We propose a novel MDC video, NeuralMDC, demonstrating how bidirectional transformers trained for masked token prediction can vastly simplify the design of MDC video.
• Score: 19.77771020475519
• License:
• Abstract: Multiple Description Coding (MDC) is a promising error-resilient source coding method that is particularly suitable for dynamic networks with multiple (yet noisy and unreliable) paths. However, conventional MDC video codecs suffer from cumbersome architectures, poor scalability, limited loss resilience, and lower compression efficiency. As a result, MDC has never been widely adopted. Inspired by the potential of neural video codecs, this paper rethinks MDC design. We propose a novel MDC video codec, NeuralMDC, demonstrating how bidirectional transformers trained for masked token prediction can vastly simplify the design of MDC video codec. To compress a video, NeuralMDC starts by tokenizing each frame into its latent representation and then splits the latent tokens to create multiple descriptions containing correlated information. Instead of using motion prediction and warping operations, NeuralMDC trains a bidirectional masked transformer to model the spatial-temporal dependencies of latent representations and predict the distribution of the current representation based on the past. The predicted distribution is used to independently entropy code each description and infer any potentially lost tokens. Extensive experiments demonstrate NeuralMDC achieves state-of-the-art loss resilience with minimal sacrifices in compression efficiency, significantly outperforming the best existing residual-coding-based error-resilient neural video codec.

Related papers

• Neural Image Compression Using Masked Sparse Visual Representation [17.229601298529825]
  We study neural image compression based on the Sparse Visual Representation (SVR), where images are embedded into a discrete latent space spanned by learned visual codebooks. By sharing codebooks with the decoder, the encoder transfers codeword indices that are efficient and cross-platform robust. We propose a Masked Adaptive Codebook learning (M-AdaCode) method that applies masks to the latent feature subspace to balance and reconstruction quality.
  arXiv  Detail & Related papers  (2023-09-20T21:59:23Z)
• VNVC: A Versatile Neural Video Coding Framework for Efficient Human-Machine Vision [59.632286735304156]
  It is more efficient to enhance/analyze the coded representations directly without decoding them into pixels. We propose a versatile neural video coding (VNVC) framework, which targets learning compact representations to support both reconstruction and direct enhancement/analysis.
  arXiv  Detail & Related papers  (2023-06-19T03:04:57Z)
• Scalable Neural Video Representations with Learnable Positional Features [73.51591757726493]
  We show how to train neural representations with learnable positional features (NVP) that effectively amortize a video as latent codes. We demonstrate the superiority of NVP on the popular UVG benchmark; compared with prior arts, NVP not only trains 2 times faster (less than 5 minutes) but also exceeds their encoding quality as 34.07rightarrow$34.57 (measured with the PSNR metric)
  arXiv  Detail & Related papers  (2022-10-13T08:15:08Z)
• Multi-stage image denoising with the wavelet transform [125.2251438120701]
  Deep convolutional neural networks (CNNs) are used for image denoising via automatically mining accurate structure information. We propose a multi-stage image denoising CNN with the wavelet transform (MWDCNN) via three stages, i.e., a dynamic convolutional block (DCB), two cascaded wavelet transform and enhancement blocks (WEBs) and residual block (RB)
  arXiv  Detail & Related papers  (2022-09-26T03:28:23Z)
• Neural Data-Dependent Transform for Learned Image Compression [72.86505042102155]
  We build a neural data-dependent transform and introduce a continuous online mode decision mechanism to jointly optimize the coding efficiency for each individual image. The experimental results show the effectiveness of the proposed neural-syntax design and the continuous online mode decision mechanism.
  arXiv  Detail & Related papers  (2022-03-09T14:56:48Z)
• Reducing Redundancy in the Bottleneck Representation of the Autoencoders [98.78384185493624]
  Autoencoders are a type of unsupervised neural networks, which can be used to solve various tasks. We propose a scheme to explicitly penalize feature redundancies in the bottleneck representation. We tested our approach across different tasks: dimensionality reduction using three different dataset, image compression using the MNIST dataset, and image denoising using fashion MNIST.
  arXiv  Detail & Related papers  (2022-02-09T18:48:02Z)
• End-to-end Neural Video Coding Using a Compound Spatiotemporal Representation [33.54844063875569]
  We propose a hybrid motion compensation (HMC) method that adaptively combines the predictions generated by two approaches. Specifically, we generate a compoundtemporal representation (STR) through a recurrent information aggregation (RIA) module. We further design a one-to-many decoder pipeline to generate multiple predictions from the CSTR, including vector-based resampling, adaptive kernel-based resampling, compensation mode selection maps and texture enhancements.
  arXiv  Detail & Related papers  (2021-08-05T19:43:32Z)
• Neural Video Coding using Multiscale Motion Compensation and Spatiotemporal Context Model [45.46660511313426]
  We propose an end-to-end deep neural video coding framework (NVC) It uses variational autoencoders (VAEs) with joint spatial and temporal prior aggregation (PA) to exploit the correlations in intra-frame pixels, inter-frame motions and inter-frame compensation residuals. NVC is evaluated for the low-delay causal settings and compared with H.265/HEVC, H.264/AVC and the other learnt video compression methods.
  arXiv  Detail & Related papers  (2020-07-09T06:15:17Z)
• Modeling Lost Information in Lossy Image Compression [72.69327382643549]
  Lossy image compression is one of the most commonly used operators for digital images. We propose a novel invertible framework called Invertible Lossy Compression (ILC) to largely mitigate the information loss problem.
  arXiv  Detail & Related papers  (2020-06-22T04:04:56Z)
• Deep Optimized Multiple Description Image Coding via Scalar Quantization Learning [37.00592782976494]
  We introduce a deep multiple description coding (MDC) framework optimized by minimizing multiple description (MD) compressive loss. An auto-encoder network composed of these two types of network is designed as a symmetrical parameter sharing structure. Our framework performs better than several state-of-the-art MDC approaches regarding image coding efficiency when tested on several commonly available datasets.
  arXiv  Detail & Related papers  (2020-01-12T05:03:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.