Differentiable Search for Finding Optimal Quantization Strategy

• URL: http://arxiv.org/abs/2404.08010v2
• Date: Mon, 15 Apr 2024 06:08:51 GMT
• Title: Differentiable Search for Finding Optimal Quantization Strategy
• Authors: Lianqiang Li, Chenqian Yan, Yefei Chen,
• Abstract summary: We propose a differentiable quantization strategy search (DQSS) to assign optimal quantization strategy for individual layer. We conduct DQSS for post-training quantization to enable their performance to be comparable with that in full precision models. We also employ DQSS in quantization-aware training for further validating the effectiveness of DQSS.
• Score: 3.295889768579819
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To accelerate and compress deep neural networks (DNNs), many network quantization algorithms have been proposed. Although the quantization strategy of any algorithm from the state-of-the-arts may outperform others in some network architectures, it is hard to prove the strategy is always better than others, and even cannot judge that the strategy is always the best choice for all layers in a network. In other words, existing quantization algorithms are suboptimal as they ignore the different characteristics of different layers and quantize all layers by a uniform quantization strategy. To solve the issue, in this paper, we propose a differentiable quantization strategy search (DQSS) to assign optimal quantization strategy for individual layer by taking advantages of the benefits of different quantization algorithms. Specifically, we formulate DQSS as a differentiable neural architecture search problem and adopt an efficient convolution to efficiently explore the mixed quantization strategies from a global perspective by gradient-based optimization. We conduct DQSS for post-training quantization to enable their performance to be comparable with that in full precision models. We also employ DQSS in quantization-aware training for further validating the effectiveness of DQSS. To circumvent the expensive optimization cost when employing DQSS in quantization-aware training, we update the hyper-parameters and the network parameters in a single forward-backward pass. Besides, we adjust the optimization process to avoid the potential under-fitting problem. Comprehensive experiments on high level computer vision task, i.e., image classification, and low level computer vision task, i.e., image super-resolution, with various network architectures show that DQSS could outperform the state-of-the-arts.

Related papers

• Quantum Local Search for Traveling Salesman Problem with Path-Slicing Strategy [1.8186826508785554]
  We present novel path-slicing strategies integrated with quantum local search to optimize solutions for the Traveling Salesman Problem (TSP) We explore various path slicing methods, including k-means and anti-k-means clustering, to divide the TSP into manageable subproblems. These are then solved using quantum or classical solvers.
  arXiv  Detail & Related papers  (2024-07-18T15:55:01Z)
• Qubit-efficient Variational Quantum Algorithms for Image Segmentation [4.737806718785056]
  Quantum computing is expected to transform a range of computational tasks beyond the reach of classical algorithms. In this work, we examine the application of variational quantum algorithms (VQAs) for unsupervised image segmentation.
  arXiv  Detail & Related papers  (2024-05-23T10:21:57Z)
• Bayesian Parameterized Quantum Circuit Optimization (BPQCO): A task and hardware-dependent approach [49.89480853499917]
  Variational quantum algorithms (VQA) have emerged as a promising quantum alternative for solving optimization and machine learning problems. In this paper, we experimentally demonstrate the influence of the circuit design on the performance obtained for two classification problems. We also study the degradation of the obtained circuits in the presence of noise when simulating real quantum computers.
  arXiv  Detail & Related papers  (2024-04-17T11:00:12Z)
• Quantum Architecture Search with Unsupervised Representation Learning [24.698519892763283]
  Unsupervised representation learning presents new opportunities for advancing Quantum Architecture Search (QAS) QAS is designed to optimize quantum circuits for Variational Quantum Algorithms (VQAs)
  arXiv  Detail & Related papers  (2024-01-21T19:53:17Z)
• Quantum-Informed Recursive Optimization Algorithms [0.0]
  We propose and implement a family of quantum-informed recursive optimization (QIRO) algorithms for optimization problems. Our approach leverages quantum resources to obtain information that is used in problem-specific classical reduction steps. We use backtracking techniques to further improve the performance of the algorithm without increasing the requirements on the quantum hardware.
  arXiv  Detail & Related papers  (2023-08-25T18:02:06Z)
• ORQVIZ: Visualizing High-Dimensional Landscapes in Variational Quantum Algorithms [51.02972483763309]
  Variational Quantum Algorithms (VQAs) are promising candidates for finding practical applications of quantum computers. This work is accompanied by the release of the open-source Python package $textitorqviz$, which provides code to compute and flexibly plot 1D and 2D scans.
  arXiv  Detail & Related papers  (2021-11-08T18:17:59Z)
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization [45.22093693422085]
  Mixed-precision quantization model can match different quantization bit-precisions according to the sensitivity of different layers to achieve great performance. It is a difficult problem to quickly determine the quantization bit-precision of each layer in deep neural networks according to some constraints. We propose a novel sequential single path search (SSPS) method for mixed-precision quantization.
  arXiv  Detail & Related papers  (2021-03-04T09:15:08Z)
• FactorizeNet: Progressive Depth Factorization for Efficient Network Architecture Exploration Under Quantization Constraints [93.4221402881609]
  We introduce a progressive depth factorization strategy for efficient CNN architecture exploration under quantization constraints. By algorithmically increasing the granularity of depth factorization in a progressive manner, the proposed strategy enables a fine-grained, low-level analysis of layer-wise distributions. Such a progressive depth factorization strategy also enables efficient identification of the optimal depth-factorized macroarchitecture design.
  arXiv  Detail & Related papers  (2020-11-30T07:12:26Z)
• Fully Quantized Image Super-Resolution Networks [81.75002888152159]
  We propose a Fully Quantized image Super-Resolution framework (FQSR) to jointly optimize efficiency and accuracy. We apply our quantization scheme on multiple mainstream super-resolution architectures, including SRResNet, SRGAN and EDSR. Our FQSR using low bits quantization can achieve on par performance compared with the full-precision counterparts on five benchmark datasets.
  arXiv  Detail & Related papers  (2020-11-29T03:53:49Z)
• Once Quantization-Aware Training: High Performance Extremely Low-bit Architecture Search [112.05977301976613]
  We propose to combine Network Architecture Search methods with quantization to enjoy the merits of the two sides. We first propose the joint training of architecture and quantization with a shared step size to acquire a large number of quantized models. Then a bit-inheritance scheme is introduced to transfer the quantized models to the lower bit, which further reduces the time cost and improves the quantization accuracy.
  arXiv  Detail & Related papers  (2020-10-09T03:52:16Z)
• Adaptive pruning-based optimization of parameterized quantum circuits [62.997667081978825]
  Variisy hybrid quantum-classical algorithms are powerful tools to maximize the use of Noisy Intermediate Scale Quantum devices. We propose a strategy for such ansatze used in variational quantum algorithms, which we call "Efficient Circuit Training" (PECT) Instead of optimizing all of the ansatz parameters at once, PECT launches a sequence of variational algorithms.
  arXiv  Detail & Related papers  (2020-10-01T18:14:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.