NIPQ: Noise proxy-based Integrated Pseudo-Quantization

• URL: http://arxiv.org/abs/2206.00820v2
• Date: Sat, 1 Jul 2023 08:27:18 GMT
• Title: NIPQ: Noise proxy-based Integrated Pseudo-Quantization
• Authors: Juncheol Shin, Junhyuk So, Sein Park, Seungyeop Kang, Sungjoo Yoo and Eunhyeok Park
• Abstract summary: Straight-through estimator (STE) incurs unstable convergence during quantization-aware training (QAT) We propose a novel noise proxy-based integrated pseudoquantization (NIPQ) that enables unified support of pseudoquantization for both activation and weight. NIPQ outperforms existing quantization algorithms in various vision and language applications by a large margin.
• Score: 9.207644534257543
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Straight-through estimator (STE), which enables the gradient flow over the non-differentiable function via approximation, has been favored in studies related to quantization-aware training (QAT). However, STE incurs unstable convergence during QAT, resulting in notable quality degradation in low precision. Recently, pseudoquantization training has been proposed as an alternative approach to updating the learnable parameters using the pseudo-quantization noise instead of STE. In this study, we propose a novel noise proxy-based integrated pseudoquantization (NIPQ) that enables unified support of pseudoquantization for both activation and weight by integrating the idea of truncation on the pseudo-quantization framework. NIPQ updates all of the quantization parameters (e.g., bit-width and truncation boundary) as well as the network parameters via gradient descent without STE instability. According to our extensive experiments, NIPQ outperforms existing quantization algorithms in various vision and language applications by a large margin.

Related papers

• Adaptive variational quantum dynamics simulations with compressed circuits and fewer measurements [4.2643127089535104]
  We show an improved version of the adaptive variational quantum dynamics simulation (AVQDS) method, which we call AVQDS(T) The algorithm adaptively adds layers of disjoint unitary gates to the ansatz circuit so as to keep the McLachlan distance, a measure of the accuracy of the variational dynamics, below a fixed threshold. We also show a method based on eigenvalue truncation to solve the linear equations of motion for the variational parameters with enhanced noise resilience.
  arXiv  Detail & Related papers  (2024-08-13T02:56:43Z)
• Quantum Shadow Gradient Descent for Variational Quantum Algorithms [14.286227676294034]
  Gradient-based gradient estimation has been proposed for training variational quantum circuits in quantum neural networks (QNNs) The task of gradient estimation has proven to be challenging due to distinctive quantum features such as state collapse and measurement incompatibility. We develop a novel procedure called quantum shadow descent that uses a single sample per iteration to estimate all components of the gradient.
  arXiv  Detail & Related papers  (2023-10-10T18:45:43Z)
• CSQ: Growing Mixed-Precision Quantization Scheme with Bi-level Continuous Sparsification [51.81850995661478]
  Mixed-precision quantization has been widely applied on deep neural networks (DNNs) Previous attempts on bit-level regularization and pruning-based dynamic precision adjustment during training suffer from noisy gradients and unstable convergence. We propose Continuous Sparsification Quantization (CSQ), a bit-level training method to search for mixed-precision quantization schemes with improved stability.
  arXiv  Detail & Related papers  (2022-12-06T05:44:21Z)
• Error Mitigation-Aided Optimization of Parameterized Quantum Circuits: Convergence Analysis [42.275148861039895]
  Variational quantum algorithms (VQAs) offer the most promising path to obtaining quantum advantages via noisy processors. gate noise due to imperfections and decoherence affects the gradient estimates by introducing a bias. Quantum error mitigation (QEM) techniques can reduce the estimation bias without requiring any increase in the number of qubits. QEM can reduce the number of required iterations, but only as long as the quantum noise level is sufficiently small.
  arXiv  Detail & Related papers  (2022-09-23T10:48:04Z)
• Symmetric Pruning in Quantum Neural Networks [111.438286016951]
  Quantum neural networks (QNNs) exert the power of modern quantum machines. QNNs with handcraft symmetric ansatzes generally experience better trainability than those with asymmetric ansatzes. We propose the effective quantum neural tangent kernel (EQNTK) to quantify the convergence of QNNs towards the global optima.
  arXiv  Detail & Related papers  (2022-08-30T08:17:55Z)
• Symmetry Regularization and Saturating Nonlinearity for Robust Quantization [5.1779694507922835]
  We present three insights to robustify a network against quantization. We propose two novel methods called symmetry regularization (SymReg) and saturating nonlinearity (SatNL) Applying the proposed methods during training can enhance the robustness of arbitrary neural networks against quantization.
  arXiv  Detail & Related papers  (2022-07-31T02:12:28Z)
• Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation [48.838691414561694]
  Nonuniform-to-Uniform Quantization (N2UQ) is a method that can maintain the strong representation ability of nonuniform methods while being hardware-friendly and efficient. N2UQ outperforms state-of-the-art nonuniform quantization methods by 0.71.8% on ImageNet.
  arXiv  Detail & Related papers  (2021-11-29T18:59:55Z)
• Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss [61.26793005355441]
  Cluster-Promoting Quantization (CPQ) finds the optimal quantization grids for neural networks. DropBits is a new bit-drop technique that revises the standard dropout regularization to randomly drop bits instead of neurons. We experimentally validate our method on various benchmark datasets and network architectures.
  arXiv  Detail & Related papers  (2021-09-05T15:15:07Z)
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks [70.77754244060384]
  Fully quantized training (FQT) uses low-bitwidth hardware by quantizing the activations, weights, and gradients of a neural network model. One major challenge with FQT is the lack of theoretical understanding, in particular of how gradient quantization impacts convergence properties.
  arXiv  Detail & Related papers  (2020-10-27T13:57:33Z)
• Variance Reduction for Deep Q-Learning using Stochastic Recursive Gradient [51.880464915253924]
  Deep Q-learning algorithms often suffer from poor gradient estimations with an excessive variance. This paper introduces the framework for updating the gradient estimates in deep Q-learning, achieving a novel algorithm called SRG-DQN.
  arXiv  Detail & Related papers  (2020-07-25T00:54:20Z)
• Policy Gradient based Quantum Approximate Optimization Algorithm [2.5614220901453333]
  We show that policy-gradient-based reinforcement learning algorithms are well suited for optimizing the variational parameters of QAOA in a noise-robust fashion. We analyze the performance of the algorithm for quantum state transfer problems in single- and multi-qubit systems.
  arXiv  Detail & Related papers  (2020-02-04T00:46:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.