VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations

• URL: http://arxiv.org/abs/2310.14487v1
• Date: Mon, 23 Oct 2023 01:41:38 GMT
• Title: VQ-NeRF: Vector Quantization Enhances Implicit Neural Representations
• Authors: Yiying Yang, Wen Liu, Fukun Yin, Xin Chen, Gang Yu, Jiayuan Fan, Tao Chen
• Abstract summary: VQ-NeRF is an efficient pipeline for enhancing implicit neural representations via vector quantization. We present an innovative multi-scale NeRF sampling scheme that concurrently optimize the NeRF model at both compressed and original scales. We incorporate a semantic loss function to improve the geometric fidelity and semantic coherence of our 3D reconstructions.
• Score: 25.88881764546414
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advancements in implicit neural representations have contributed to high-fidelity surface reconstruction and photorealistic novel view synthesis. However, the computational complexity inherent in these methodologies presents a substantial impediment, constraining the attainable frame rates and resolutions in practical applications. In response to this predicament, we propose VQ-NeRF, an effective and efficient pipeline for enhancing implicit neural representations via vector quantization. The essence of our method involves reducing the sampling space of NeRF to a lower resolution and subsequently reinstating it to the original size utilizing a pre-trained VAE decoder, thereby effectively mitigating the sampling time bottleneck encountered during rendering. Although the codebook furnishes representative features, reconstructing fine texture details of the scene remains challenging due to high compression rates. To overcome this constraint, we design an innovative multi-scale NeRF sampling scheme that concurrently optimizes the NeRF model at both compressed and original scales to enhance the network's ability to preserve fine details. Furthermore, we incorporate a semantic loss function to improve the geometric fidelity and semantic coherence of our 3D reconstructions. Extensive experiments demonstrate the effectiveness of our model in achieving the optimal trade-off between rendering quality and efficiency. Evaluation on the DTU, BlendMVS, and H3DS datasets confirms the superior performance of our approach.

Related papers

• Magnituder Layers for Implicit Neural Representations in 3D [23.135779936528333]
  We introduce a novel neural network layer called the "magnituder" By integrating magnituders into standard feed-forward layer stacks, we achieve improved inference speed and adaptability. Our approach enables a zero-shot performance boost in trained implicit neural representation models.
  arXiv  Detail & Related papers  (2024-10-13T08:06:41Z)
• Neural NeRF Compression [19.853882143024]
  Recent NeRFs utilize feature grids to improve rendering quality and speed. These representations introduce significant storage overhead. This paper presents a novel method for efficiently compressing a grid-based NeRF model.
  arXiv  Detail & Related papers  (2024-06-13T09:12:26Z)
• Spatial Annealing Smoothing for Efficient Few-shot Neural Rendering [106.0057551634008]
  We introduce an accurate and efficient few-shot neural rendering method named Spatial Annealing smoothing regularized NeRF (SANeRF) By adding merely one line of code, SANeRF delivers superior rendering quality and much faster reconstruction speed compared to current few-shot NeRF methods.
  arXiv  Detail & Related papers  (2024-06-12T02:48:52Z)
• NeRF-DetS: Enhancing Multi-View 3D Object Detection with Sampling-adaptive Network of Continuous NeRF-based Representation [60.47114985993196]
  NeRF-Det unifies the tasks of novel view arithmetic and 3D perception. We introduce a novel 3D perception network structure, NeRF-DetS. NeRF-DetS outperforms competitive NeRF-Det on the ScanNetV2 dataset.
  arXiv  Detail & Related papers  (2024-04-22T06:59:03Z)
• Neural Poisson Surface Reconstruction: Resolution-Agnostic Shape Reconstruction from Point Clouds [53.02191521770926]
  We introduce Neural Poisson Surface Reconstruction (nPSR), an architecture for shape reconstruction that addresses the challenge of recovering 3D shapes from points. nPSR exhibits two main advantages: First, it enables efficient training on low-resolution data while achieving comparable performance at high-resolution evaluation. Overall, the neural Poisson surface reconstruction not only improves upon the limitations of classical deep neural networks in shape reconstruction but also achieves superior results in terms of reconstruction quality, running time, and resolution agnosticism.
  arXiv  Detail & Related papers  (2023-08-03T13:56:07Z)
• From NeRFLiX to NeRFLiX++: A General NeRF-Agnostic Restorer Paradigm [57.73868344064043]
  We propose NeRFLiX, a general NeRF-agnostic restorer paradigm that learns a degradation-driven inter-viewpoint mixer. We also present NeRFLiX++ with a stronger two-stage NeRF degradation simulator and a faster inter-viewpoint mixer. NeRFLiX++ is capable of restoring photo-realistic ultra-high-resolution outputs from noisy low-resolution NeRF-rendered views.
  arXiv  Detail & Related papers  (2023-06-10T09:19:19Z)
• Improved Quasi-Recurrent Neural Network for Hyperspectral Image Denoising [9.723155514555765]
  We show that with a few simple modifications, the performance of QRNN3D could be substantially improved further. We introduce an adaptive fusion module to replace its vanilla additive skip connection to better fuse the features of the encoder and decoder. Experimental results on various noise settings demonstrate the effectiveness and superior performance of our method.
  arXiv  Detail & Related papers  (2022-11-27T12:38:03Z)
• Implicit Neural Representation Learning for Hyperspectral Image Super-Resolution [0.0]
  Implicit Neural Representations (INRs) are making strides as a novel and effective representation. We propose a novel HSI reconstruction model based on INR which represents HSI by a continuous function mapping a spatial coordinate to its corresponding spectral radiance values.
  arXiv  Detail & Related papers  (2021-12-20T14:07:54Z)
• NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo [97.07453889070574]
  We present a new multi-view depth estimation method that utilizes both conventional SfM reconstruction and learning-based priors. We show that our proposed framework significantly outperforms state-of-the-art methods on indoor scenes.
  arXiv  Detail & Related papers  (2021-09-02T17:54:31Z)
• NeRF in detail: Learning to sample for view synthesis [104.75126790300735]
  Neural radiance fields (NeRF) methods have demonstrated impressive novel view synthesis. In this work we address a clear limitation of the vanilla coarse-to-fine approach -- that it is based on a performance and not trained end-to-end for the task at hand. We introduce a differentiable module that learns to propose samples and their importance for the fine network, and consider and compare multiple alternatives for its neural architecture.
  arXiv  Detail & Related papers  (2021-06-09T17:59:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.