On Quantizing Implicit Neural Representations

• URL: http://arxiv.org/abs/2209.01019v1
• Date: Thu, 1 Sep 2022 05:48:37 GMT
• Title: On Quantizing Implicit Neural Representations
• Authors: Cameron Gordon, Shin-Fang Chng, Lachlan MacDonald, Simon Lucey
• Abstract summary: We show that a non-uniform quantization of neural weights can lead to significant improvements. We demonstrate that it is possible (while memory inefficient) to reconstruct signals using binary neural networks.
• Score: 30.257625048084968
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The role of quantization within implicit/coordinate neural networks is still not fully understood. We note that using a canonical fixed quantization scheme during training produces poor performance at low-rates due to the network weight distributions changing over the course of training. In this work, we show that a non-uniform quantization of neural weights can lead to significant improvements. Specifically, we demonstrate that a clustered quantization enables improved reconstruction. Finally, by characterising a trade-off between quantization and network capacity, we demonstrate that it is possible (while memory inefficient) to reconstruct signals using binary neural networks. We demonstrate our findings experimentally on 2D image reconstruction and 3D radiance fields; and show that simple quantization methods and architecture search can achieve compression of NeRF to less than 16kb with minimal loss in performance (323x smaller than the original NeRF).

Related papers

• Convolutional Neural Networks Quantization with Attention [1.0312968200748118]
  We propose a method, double-stage Squeeze-and-Threshold (double-stage ST) It uses the attention mechanism to quantize networks and achieve state-of-art results.
  arXiv  Detail & Related papers  (2022-09-30T08:48:31Z)
• BiTAT: Neural Network Binarization with Task-dependent Aggregated Transformation [116.26521375592759]
  Quantization aims to transform high-precision weights and activations of a given neural network into low-precision weights/activations for reduced memory usage and computation. Extreme quantization (1-bit weight/1-bit activations) of compactly-designed backbone architectures results in severe performance degeneration. This paper proposes a novel Quantization-Aware Training (QAT) method that can effectively alleviate performance degeneration.
  arXiv  Detail & Related papers  (2022-07-04T13:25:49Z)
• Post-training Quantization for Neural Networks with Provable Guarantees [9.58246628652846]
  We modify a post-training neural-network quantization method, GPFQ, that is based on a greedy path-following mechanism. We prove that for quantizing a single-layer network, the relative square error essentially decays linearly in the number of weights.
  arXiv  Detail & Related papers  (2022-01-26T18:47:38Z)
• Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss [61.26793005355441]
  Cluster-Promoting Quantization (CPQ) finds the optimal quantization grids for neural networks. DropBits is a new bit-drop technique that revises the standard dropout regularization to randomly drop bits instead of neurons. We experimentally validate our method on various benchmark datasets and network architectures.
  arXiv  Detail & Related papers  (2021-09-05T15:15:07Z)
• MWQ: Multiscale Wavelet Quantized Neural Networks [45.22093693422084]
  We propose a novel multiscale wavelet quantization (MWQ) method inspired by the characteristics of images in the frequency domain. It exploits the multiscale frequency and spatial information to alleviate the information loss caused by quantization in the spatial domain. Because of the flexibility of MWQ, we demonstrate three applications on the ImageNet and COCO datasets.
  arXiv  Detail & Related papers  (2021-03-09T11:21:59Z)
• Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks [73.29587731448345]
  This paper proposes two novel techniques to train deep convolutional neural networks with low bit-width weights and activations. First, to obtain low bit-width weights, most existing methods obtain the quantized weights by performing quantization on the full-precision network weights. Second, to obtain low bit-width activations, existing works consider all channels equally.
  arXiv  Detail & Related papers  (2020-12-26T15:21:18Z)
• DAQ: Distribution-Aware Quantization for Deep Image Super-Resolution Networks [49.191062785007006]
  Quantizing deep convolutional neural networks for image super-resolution substantially reduces their computational costs. Existing works either suffer from a severe performance drop in ultra-low precision of 4 or lower bit-widths, or require a heavy fine-tuning process to recover the performance. We propose a novel distribution-aware quantization scheme (DAQ) which facilitates accurate training-free quantization in ultra-low precision.
  arXiv  Detail & Related papers  (2020-12-21T10:19:42Z)
• Recurrence of Optimum for Training Weight and Activation Quantized Networks [4.103701929881022]
  Training deep learning models with low-precision weights and activations involves a demanding optimization task. We show how to overcome the nature of network quantization. We also show numerical evidence of the recurrence phenomenon of weight evolution in training quantized deep networks.
  arXiv  Detail & Related papers  (2020-12-10T09:14:43Z)
• A Greedy Algorithm for Quantizing Neural Networks [4.683806391173103]
  We propose a new computationally efficient method for quantizing the weights of pre- trained neural networks. Our method deterministically quantizes layers in an iterative fashion with no complicated re-training required.
  arXiv  Detail & Related papers  (2020-10-29T22:53:10Z)
• Searching for Low-Bit Weights in Quantized Neural Networks [129.8319019563356]
  Quantized neural networks with low-bit weights and activations are attractive for developing AI accelerators. We present to regard the discrete weights in an arbitrary quantized neural network as searchable variables, and utilize a differential method to search them accurately.
  arXiv  Detail & Related papers  (2020-09-18T09:13:26Z)
• Compressive sensing with un-trained neural networks: Gradient descent finds the smoothest approximation [60.80172153614544]
  Un-trained convolutional neural networks have emerged as highly successful tools for image recovery and restoration. We show that an un-trained convolutional neural network can approximately reconstruct signals and images that are sufficiently structured, from a near minimal number of random measurements.
  arXiv  Detail & Related papers  (2020-05-07T15:57:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.