NERV++: An Enhanced Implicit Neural Video Representation

• URL: http://arxiv.org/abs/2402.18305v1
• Date: Wed, 28 Feb 2024 13:00:32 GMT
• Title: NERV++: An Enhanced Implicit Neural Video Representation
• Authors: Ahmed Ghorbel, Wassim Hamidouche, Luce Morin
• Abstract summary: We introduce neural representations for videos NeRV++, an enhanced implicit neural video representation. NeRV++ is more straightforward yet effective enhancement over the original NeRV decoder architecture. We evaluate our method on UVG, MCL JVC, and Bunny datasets, achieving competitive results for video compression with INRs.
• Score: 11.25130799452367
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural fields, also known as implicit neural representations (INRs), have shown a remarkable capability of representing, generating, and manipulating various data types, allowing for continuous data reconstruction at a low memory footprint. Though promising, INRs applied to video compression still need to improve their rate-distortion performance by a large margin, and require a huge number of parameters and long training iterations to capture high-frequency details, limiting their wider applicability. Resolving this problem remains a quite challenging task, which would make INRs more accessible in compression tasks. We take a step towards resolving these shortcomings by introducing neural representations for videos NeRV++, an enhanced implicit neural video representation, as more straightforward yet effective enhancement over the original NeRV decoder architecture, featuring separable conv2d residual blocks (SCRBs) that sandwiches the upsampling block (UB), and a bilinear interpolation skip layer for improved feature representation. NeRV++ allows videos to be directly represented as a function approximated by a neural network, and significantly enhance the representation capacity beyond current INR-based video codecs. We evaluate our method on UVG, MCL JVC, and Bunny datasets, achieving competitive results for video compression with INRs. This achievement narrows the gap to autoencoder-based video coding, marking a significant stride in INR-based video compression research.

Related papers

• NVRC: Neural Video Representation Compression [13.131842990481038]
  We propose a novel INR-based video compression framework, Neural Video Representation Compression (NVRC) NVRC, for the first time, is able to optimize an INR-based video in a fully end-to-end manner. Our experiments show that NVRC outperforms many conventional and learning-based benchmark entropy.
  arXiv  Detail & Related papers  (2024-09-11T16:57:12Z)
• VQ-NeRV: A Vector Quantized Neural Representation for Videos [3.6662666629446043]
  Implicit neural representations (INR) excel in encoding videos within neural networks, showcasing promise in computer vision tasks like video compression and denoising. We introduce an advanced U-shaped architecture, Vector Quantized-NeRV (VQ-NeRV), which integrates a novel component--the VQ-NeRV Block. This block incorporates a codebook mechanism to discretize the network's shallow residual features and inter-frame residual information effectively.
  arXiv  Detail & Related papers  (2024-03-19T03:19:07Z)
• Boosting Neural Representations for Videos with a Conditional Decoder [28.073607937396552]
  Implicit neural representations (INRs) have emerged as a promising approach for video storage and processing. This paper introduces a universal boosting framework for current implicit video representation approaches.
  arXiv  Detail & Related papers  (2024-02-28T08:32:19Z)
• Progressive Fourier Neural Representation for Sequential Video Compilation [75.43041679717376]
  Motivated by continual learning, this work investigates how to accumulate and transfer neural implicit representations for multiple complex video data over sequential encoding sessions. We propose a novel method, Progressive Fourier Neural Representation (PFNR), that aims to find an adaptive and compact sub-module in Fourier space to encode videos in each training session. We validate our PFNR method on the UVG8/17 and DAVIS50 video sequence benchmarks and achieve impressive performance gains over strong continual learning baselines.
  arXiv  Detail & Related papers  (2023-06-20T06:02:19Z)
• HiNeRV: Video Compression with Hierarchical Encoding-based Neural Representation [14.088444622391501]
  Implicit Representations (INRs) have previously been used to represent and compress image and video content. Existing INR-based methods have failed to deliver rate quality performance comparable with the state of the art in video compression. We propose HiNeRV, an INR that combines light weight layers with hierarchical positional encodings.
  arXiv  Detail & Related papers  (2023-06-16T12:59:52Z)
• HNeRV: A Hybrid Neural Representation for Videos [56.492309149698606]
  Implicit neural representations store videos as neural networks. We propose a Hybrid Neural Representation for Videos (HNeRV) With content-adaptive embeddings and re-designed architecture, HNeRV outperforms implicit methods in video regression tasks.
  arXiv  Detail & Related papers  (2023-04-05T17:55:04Z)
• Towards Scalable Neural Representation for Diverse Videos [68.73612099741956]
  Implicit neural representations (INR) have gained increasing attention in representing 3D scenes and images. Existing INR-based methods are limited to encoding a handful of short videos with redundant visual content. This paper focuses on developing neural representations for encoding long and/or a large number of videos with diverse visual content.
  arXiv  Detail & Related papers  (2023-03-24T16:32:19Z)
• Modality-Agnostic Variational Compression of Implicit Neural Representations [96.35492043867104]
  We introduce a modality-agnostic neural compression algorithm based on a functional view of data and parameterised as an Implicit Neural Representation (INR) Bridging the gap between latent coding and sparsity, we obtain compact latent representations non-linearly mapped to a soft gating mechanism. After obtaining a dataset of such latent representations, we directly optimise the rate/distortion trade-off in a modality-agnostic space using neural compression.
  arXiv  Detail & Related papers  (2023-01-23T15:22:42Z)
• CNeRV: Content-adaptive Neural Representation for Visual Data [54.99373641890767]
  We propose Neural Visual Representation with Content-adaptive Embedding (CNeRV), which combines the generalizability of autoencoders with the simplicity and compactness of implicit representation. We match the performance of NeRV, a state-of-the-art implicit neural representation, on the reconstruction task for frames seen during training while far surpassing for frames that are skipped during training (unseen images) With the same latent code length and similar model size, CNeRV outperforms autoencoders on reconstruction of both seen and unseen images.
  arXiv  Detail & Related papers  (2022-11-18T18:35:43Z)
• Scalable Neural Video Representations with Learnable Positional Features [73.51591757726493]
  We show how to train neural representations with learnable positional features (NVP) that effectively amortize a video as latent codes. We demonstrate the superiority of NVP on the popular UVG benchmark; compared with prior arts, NVP not only trains 2 times faster (less than 5 minutes) but also exceeds their encoding quality as 34.07rightarrow$34.57 (measured with the PSNR metric)
  arXiv  Detail & Related papers  (2022-10-13T08:15:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.