Neural Compression-Based Feature Learning for Video Restoration

• URL: http://arxiv.org/abs/2203.09208v2
• Date: Fri, 18 Mar 2022 05:10:12 GMT
• Title: Neural Compression-Based Feature Learning for Video Restoration
• Authors: Cong Huang and Jiahao Li and Bin Li and Dong Liu and Yan Lu
• Abstract summary: This paper proposes learning noise-robust feature representations to help video restoration. We design a neural compression module to filter the noise and keep the most useful information in features for video restoration.
• Score: 29.021502115116736
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How to efficiently utilize the temporal features is crucial, yet challenging, for video restoration. The temporal features usually contain various noisy and uncorrelated information, and they may interfere with the restoration of the current frame. This paper proposes learning noise-robust feature representations to help video restoration. We are inspired by that the neural codec is a natural denoiser. In neural codec, the noisy and uncorrelated contents which are hard to predict but cost lots of bits are more inclined to be discarded for bitrate saving. Therefore, we design a neural compression module to filter the noise and keep the most useful information in features for video restoration. To achieve robustness to noise, our compression module adopts a spatial channel-wise quantization mechanism to adaptively determine the quantization step size for each position in the latent. Experiments show that our method can significantly boost the performance on video denoising, where we obtain 0.13 dB improvement over BasicVSR++ with only 0.23x FLOPs. Meanwhile, our method also obtains SOTA results on video deraining and dehazing.

Related papers

• LLVD: LSTM-based Explicit Motion Modeling in Latent Space for Blind Video Denoising [1.9253333342733672]
  This paper introduces a novel algorithm designed for scenarios where noise is introduced during video capture. We propose the Latent space LSTM Video Denoiser (LLVD), an end-to-end blind denoising model. Experiments reveal that LLVD demonstrates excellent performance for both synthetic and captured noise.
  arXiv  Detail & Related papers  (2025-01-10T06:20:27Z)
• Large Motion Video Autoencoding with Cross-modal Video VAE [52.13379965800485]
  Video Variational Autoencoder (VAE) is essential for reducing video redundancy and facilitating efficient video generation. Existing Video VAEs have begun to address temporal compression; however, they often suffer from inadequate reconstruction performance. We present a novel and powerful video autoencoder capable of high-fidelity video encoding.
  arXiv  Detail & Related papers  (2024-12-23T18:58:24Z)
• VQ-NeRV: A Vector Quantized Neural Representation for Videos [3.6662666629446043]
  Implicit neural representations (INR) excel in encoding videos within neural networks, showcasing promise in computer vision tasks like video compression and denoising. We introduce an advanced U-shaped architecture, Vector Quantized-NeRV (VQ-NeRV), which integrates a novel component--the VQ-NeRV Block. This block incorporates a codebook mechanism to discretize the network's shallow residual features and inter-frame residual information effectively.
  arXiv  Detail & Related papers  (2024-03-19T03:19:07Z)
• NERV++: An Enhanced Implicit Neural Video Representation [11.25130799452367]
  We introduce neural representations for videos NeRV++, an enhanced implicit neural video representation. NeRV++ is more straightforward yet effective enhancement over the original NeRV decoder architecture. We evaluate our method on UVG, MCL JVC, and Bunny datasets, achieving competitive results for video compression with INRs.
  arXiv  Detail & Related papers  (2024-02-28T13:00:32Z)
• VCISR: Blind Single Image Super-Resolution with Video Compression Synthetic Data [18.877077302923713]
  We present a video compression-based degradation model to synthesize low-resolution image data in the blind SISR task. Our proposed image synthesizing method is widely applicable to existing image datasets. By introducing video coding artifacts to SISR degradation models, neural networks can super-resolve images with the ability to restore video compression degradations.
  arXiv  Detail & Related papers  (2023-11-02T05:24:19Z)
• High Fidelity Neural Audio Compression [92.4812002532009]
  We introduce a state-of-the-art real-time, high-fidelity, audio leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary.
  arXiv  Detail & Related papers  (2022-10-24T17:52:02Z)
• Scalable Neural Video Representations with Learnable Positional Features [73.51591757726493]
  We show how to train neural representations with learnable positional features (NVP) that effectively amortize a video as latent codes. We demonstrate the superiority of NVP on the popular UVG benchmark; compared with prior arts, NVP not only trains 2 times faster (less than 5 minutes) but also exceeds their encoding quality as 34.07rightarrow$34.57 (measured with the PSNR metric)
  arXiv  Detail & Related papers  (2022-10-13T08:15:08Z)
• Exploring Long- and Short-Range Temporal Information for Learned Video Compression [54.91301930491466]
  We focus on exploiting the unique characteristics of video content and exploring temporal information to enhance compression performance. For long-range temporal information exploitation, we propose temporal prior that can update continuously within the group of pictures (GOP) during inference. In that case temporal prior contains valuable temporal information of all decoded images within the current GOP. In detail, we design a hierarchical structure to achieve multi-scale compensation.
  arXiv  Detail & Related papers  (2022-08-07T15:57:18Z)
• Self-Conditioned Probabilistic Learning of Video Rescaling [70.10092286301997]
  We propose a self-conditioned probabilistic framework for video rescaling to learn the paired downscaling and upscaling procedures simultaneously. We decrease the entropy of the information lost in the downscaling by maximizing its conditioned probability on the strong spatial-temporal prior information. We extend the framework to a lossy video compression system, in which a gradient estimator for non-differential industrial lossy codecs is proposed.
  arXiv  Detail & Related papers  (2021-07-24T15:57:15Z)
• Content Adaptive and Error Propagation Aware Deep Video Compression [110.31693187153084]
  We propose a content adaptive and error propagation aware video compression system. Our method employs a joint training strategy by considering the compression performance of multiple consecutive frames instead of a single frame. Instead of using the hand-crafted coding modes in the traditional compression systems, we design an online encoder updating scheme in our system.
  arXiv  Detail & Related papers  (2020-03-25T09:04:24Z)
• Restore from Restored: Video Restoration with Pseudo Clean Video [28.057705167363327]
  We propose a self-supervised video denoising method called "restore-from-restored" This method fine-tunes a pre-trained network by using a pseudo clean video during the test phase. We analyze the restoration performance of the fine-tuned video denoising networks with the proposed self-supervision-based learning algorithm.
  arXiv  Detail & Related papers  (2020-03-09T17:37:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.