CodecNeRF: Toward Fast Encoding and Decoding, Compact, and High-quality Novel-view Synthesis

• URL: http://arxiv.org/abs/2404.04913v2
• Date: Tue, 28 May 2024 04:30:35 GMT
• Title: CodecNeRF: Toward Fast Encoding and Decoding, Compact, and High-quality Novel-view Synthesis
• Authors: Gyeongjin Kang, Younggeun Lee, Seungjun Oh, Eunbyung Park,
• Abstract summary: We present CodecNeRF, a neural encoder and decoder architecture that can generate a NeRF representation in a single forward pass. We also develop a novel finetuning method to efficiently adapt the generated NeRF representations to a new test instance. The proposed CodecNeRF achieved unprecedented compression performance of more than 150x and 20x reduction in encoding time.
• Score: 2.7463268699570134
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural Radiance Fields (NeRF) have achieved huge success in effectively capturing and representing 3D objects and scenes. However, several factors have impeded its further proliferation as next-generation 3D media. To establish a ubiquitous presence in everyday media formats, such as images and videos, it is imperative to devise a solution that effectively fulfills three key objectives: fast encoding and decoding time, compact model sizes, and high-quality renderings. Despite significant advancements, a comprehensive algorithm that adequately addresses all objectives has yet to be fully realized. In this work, we present CodecNeRF, a neural codec for NeRF representations, consisting of a novel encoder and decoder architecture that can generate a NeRF representation in a single forward pass. Furthermore, inspired by the recent parameter-efficient finetuning approaches, we develop a novel finetuning method to efficiently adapt the generated NeRF representations to a new test instance, leading to high-quality image renderings and compact code sizes. The proposed CodecNeRF, a newly suggested encoding-decoding-finetuning pipeline for NeRF, achieved unprecedented compression performance of more than 150x and 20x reduction in encoding time while maintaining (or improving) the image quality on widely used 3D object datasets, such as ShapeNet and Objaverse.

Related papers

• Evaluation of strategies for efficient rate-distortion NeRF streaming [4.117347527143616]
  Neural Radiance Fields (NeRF) have revolutionized the field of 3D visual representation by enabling highly realistic and detailed scene reconstructions from a sparse set of images. Despite its advancements, the efficient streaming of NeRF content remains a significant challenge due to the large amount of data involved. This paper investigates the rate-distortion performance of two NeRF streaming strategies: pixel-based and neural network (NN) parameter-based streaming.
  arXiv  Detail & Related papers  (2024-10-25T10:40:03Z)
• Neural NeRF Compression [19.853882143024]
  Recent NeRFs utilize feature grids to improve rendering quality and speed. These representations introduce significant storage overhead. This paper presents a novel method for efficiently compressing a grid-based NeRF model.
  arXiv  Detail & Related papers  (2024-06-13T09:12:26Z)
• NeRFCodec: Neural Feature Compression Meets Neural Radiance Fields for Memory-Efficient Scene Representation [22.151167286623416]
  We propose an end-to-end NeRF compression framework that integrates non-linear transform, quantization, and entropy coding for memory-efficient scene representation. We demonstrate our method outperforms existing NeRF compression methods, enabling high-quality novel view synthesis with a memory budget of 0.5 MB.
  arXiv  Detail & Related papers  (2024-04-02T15:49:00Z)
• Hyb-NeRF: A Multiresolution Hybrid Encoding for Neural Radiance Fields [12.335934855851486]
  We present Hyb-NeRF, a novel neural radiance field with a multi-resolution hybrid encoding. We show that Hyb-NeRF achieves faster rendering speed with better rending quality and even a lower memory footprint in comparison to previous methods.
  arXiv  Detail & Related papers  (2023-11-21T10:01:08Z)
• Real-time Neural Radiance Talking Portrait Synthesis via Audio-spatial Decomposition [61.6677901687009]
  We propose an efficient NeRF-based framework that enables real-time synthesizing of talking portraits. Our method can generate realistic and audio-lips synchronized talking portrait videos.
  arXiv  Detail & Related papers  (2022-11-22T16:03:11Z)
• 3D-Aware Encoding for Style-based Neural Radiance Fields [50.118687869198716]
  We learn an inversion function to project an input image to the latent space of a NeRF generator and then synthesize novel views of the original image based on the latent code. Compared with GAN inversion for 2D generative models, NeRF inversion not only needs to 1) preserve the identity of the input image, but also 2) ensure 3D consistency in generated novel views. We propose a two-stage encoder for style-based NeRF inversion.
  arXiv  Detail & Related papers  (2022-11-12T06:14:12Z)
• Scalable Neural Video Representations with Learnable Positional Features [73.51591757726493]
  We show how to train neural representations with learnable positional features (NVP) that effectively amortize a video as latent codes. We demonstrate the superiority of NVP on the popular UVG benchmark; compared with prior arts, NVP not only trains 2 times faster (less than 5 minutes) but also exceeds their encoding quality as 34.07rightarrow$34.57 (measured with the PSNR metric)
  arXiv  Detail & Related papers  (2022-10-13T08:15:08Z)
• Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields [49.41982694533966]
  We introduce a new task, Semantic-to-NeRF translation, conditioned on one single-view semantic mask as input. In particular, Sem2NeRF addresses the highly challenging task by encoding the semantic mask into the latent code that controls the 3D scene representation of a pretrained decoder. We verify the efficacy of the proposed Sem2NeRF and demonstrate it outperforms several strong baselines on two benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-21T09:15:58Z)
• Reducing Redundancy in the Bottleneck Representation of the Autoencoders [98.78384185493624]
  Autoencoders are a type of unsupervised neural networks, which can be used to solve various tasks. We propose a scheme to explicitly penalize feature redundancies in the bottleneck representation. We tested our approach across different tasks: dimensionality reduction using three different dataset, image compression using the MNIST dataset, and image denoising using fashion MNIST.
  arXiv  Detail & Related papers  (2022-02-09T18:48:02Z)
• Dynamic Neural Representational Decoders for High-Resolution Semantic Segmentation [98.05643473345474]
  We propose a novel decoder, termed dynamic neural representational decoder (NRD) As each location on the encoder's output corresponds to a local patch of the semantic labels, in this work, we represent these local patches of labels with compact neural networks. This neural representation enables our decoder to leverage the smoothness prior in the semantic label space, and thus makes our decoder more efficient.
  arXiv  Detail & Related papers  (2021-07-30T04:50:56Z)
• Recursive-NeRF: An Efficient and Dynamically Growing NeRF [34.768382663711705]
  Recursive-NeRF is an efficient rendering and training approach for the Neural Radiance Field (NeRF) method. Recursive-NeRF learns uncertainties for query coordinates, representing the quality of the predicted color and volumetric intensity at each level. Our evaluation on three public datasets shows that Recursive-NeRF is more efficient than NeRF while providing state-of-the-art quality.
  arXiv  Detail & Related papers  (2021-05-19T12:51:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.