1-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bit

• URL: http://arxiv.org/abs/2408.14267v1
• Date: Mon, 26 Aug 2024 13:42:43 GMT
• Title: 1-Bit FQT: Pushing the Limit of Fully Quantized Training to 1-bit
• Authors: Chang Gao, Jianfei Chen, Kang Zhao, Jiaqi Wang, Liping Jing,
• Abstract summary: Fully quantized training (FQT) accelerates the training of deep neural networks by quantizing the activations, weights, and gradients into lower precision. We make a first attempt to 1-bit FQT to explore the ultimate limit of FQT (the lowest achievable precision)
• Score: 41.993927897814785
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fully quantized training (FQT) accelerates the training of deep neural networks by quantizing the activations, weights, and gradients into lower precision. To explore the ultimate limit of FQT (the lowest achievable precision), we make a first attempt to 1-bit FQT. We provide a theoretical analysis of FQT based on Adam and SGD, revealing that the gradient variance influences the convergence of FQT. Building on these theoretical results, we introduce an Activation Gradient Pruning (AGP) strategy. The strategy leverages the heterogeneity of gradients by pruning less informative gradients and enhancing the numerical precision of remaining gradients to mitigate gradient variance. Additionally, we propose Sample Channel joint Quantization (SCQ), which utilizes different quantization strategies in the computation of weight gradients and activation gradients to ensure that the method is friendly to low-bitwidth hardware. Finally, we present a framework to deploy our algorithm. For fine-tuning VGGNet-16 and ResNet-18 on multiple datasets, our algorithm achieves an average accuracy improvement of approximately 6%, compared to per-sample quantization. Moreover, our training speedup can reach a maximum of 5.13x compared to full precision training.

Related papers

• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• MetaGrad: Adaptive Gradient Quantization with Hypernetworks [46.55625589293897]
  Quantization aware Training (QAT) accelerates the forward pass during the neural network training and inference. In this work, we propose to solve this problem by incorporating the gradients into the computation graph of the next training via a hypernetwork. Various experiments on CIFAR-10 dataset with different CNN network architectures demonstrate that our hypernetwork-based approach can effectively reduce the negative effect of gradient quantization noise.
  arXiv  Detail & Related papers  (2023-03-04T07:26:34Z)
• Automatic Network Adaptation for Ultra-Low Uniform-Precision Quantization [6.1664476076961146]
  Uniform-precision neural network quantization has gained popularity since it simplifies densely packed arithmetic unit for high computing capability. It ignores heterogeneous sensitivity to the impact of quantization errors across the layers, resulting in sub-optimal inference. This work proposes a novel neural architecture search called neural channel expansion that adjusts the network structure to alleviate accuracy degradation from ultra-low uniform-precision quantization.
  arXiv  Detail & Related papers  (2022-12-21T09:41:25Z)
• Quantized Training of Gradient Boosting Decision Trees [84.97123593657584]
  We propose to quantize all the high-precision gradients in a very simple yet effective way in the GBDT's training algorithm. With low-precision gradients, most arithmetic operations in GBDT training can be replaced by integer operations of 8, 16, or 32 bits. We observe up to 2$times$ speedup of our simple quantization strategy compared with SOTA GBDT systems on extensive datasets.
  arXiv  Detail & Related papers  (2022-07-20T06:27:06Z)
• LG-LSQ: Learned Gradient Linear Symmetric Quantization [3.6816597150770387]
  Deep neural networks with lower precision weights have advantages in terms of the cost of memory space and accelerator power. The main challenge associated with the quantization algorithm is maintaining accuracy at low bit-widths. We propose learned gradient linear symmetric quantization (LG-LSQ) as a method for quantizing weights and activation functions to low bit-widths.
  arXiv  Detail & Related papers  (2022-02-18T03:38:12Z)
• Distribution Adaptive INT8 Quantization for Training CNNs [12.708068468737286]
  In this paper, we propose a novel INT8 quantization training framework for convolutional neural network. Specifically, we adopt Gradient Vectorized Quantization to quantize the gradient, based on the observation that layer-wise gradients contain multiple distributions along the channel dimension. Then, Magnitude-aware Clipping Strategy is introduced by taking the magnitudes of gradients into consideration when minimizing the quantization error.
  arXiv  Detail & Related papers  (2021-02-09T11:58:10Z)
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks [70.77754244060384]
  Fully quantized training (FQT) uses low-bitwidth hardware by quantizing the activations, weights, and gradients of a neural network model. One major challenge with FQT is the lack of theoretical understanding, in particular of how gradient quantization impacts convergence properties.
  arXiv  Detail & Related papers  (2020-10-27T13:57:33Z)
• Towards Unified INT8 Training for Convolutional Neural Network [83.15673050981624]
  We build a unified 8-bit (INT8) training framework for common convolutional neural networks. First, we empirically find the four distinctive characteristics of gradients, which provide us insightful clues for gradient quantization. We propose two universal techniques, including Direction Sensitive Gradient Clipping that reduces the direction deviation of gradients.
  arXiv  Detail & Related papers  (2019-12-29T08:37:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.