End-to-end Keyword Spotting using Neural Architecture Search and Quantization

• URL: http://arxiv.org/abs/2104.06666v1
• Date: Wed, 14 Apr 2021 07:22:22 GMT
• Title: End-to-end Keyword Spotting using Neural Architecture Search and Quantization
• Authors: David Peter, Wolfgang Roth, Franz Pernkopf
• Abstract summary: This paper introduces neural architecture search (NAS) for the automatic discovery of end-to-end keyword spotting (KWS) models. We employ a differentiable NAS approach to optimize the structure of convolutional neural networks (CNNs) operating on raw audio waveforms.
• Score: 23.850887499271842
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces neural architecture search (NAS) for the automatic discovery of end-to-end keyword spotting (KWS) models in limited resource environments. We employ a differentiable NAS approach to optimize the structure of convolutional neural networks (CNNs) operating on raw audio waveforms. After a suitable KWS model is found with NAS, we conduct quantization of weights and activations to reduce the memory footprint. We conduct extensive experiments on the Google speech commands dataset. In particular, we compare our end-to-end approach to mel-frequency cepstral coefficient (MFCC) based systems. For quantization, we compare fixed bit-width quantization and trained bit-width quantization. Using NAS only, we were able to obtain a highly efficient model with an accuracy of 95.55% using 75.7k parameters and 13.6M operations. Using trained bit-width quantization, the same model achieves a test accuracy of 93.76% while using on average only 2.91 bits per activation and 2.51 bits per weight.

Related papers

• DNA Family: Boosting Weight-Sharing NAS with Block-Wise Supervisions [121.05720140641189]
  We develop a family of models with the distilling neural architecture (DNA) techniques. Our proposed DNA models can rate all architecture candidates, as opposed to previous works that can only access a sub- search space using algorithms. Our models achieve state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile convolutional network and a small vision transformer, respectively.
  arXiv  Detail & Related papers  (2024-03-02T22:16:47Z)
• BOMP-NAS: Bayesian Optimization Mixed Precision NAS [2.53488567499651]
  BOMP-NAS is able to find neural networks that achieve state of the art performance at much lower design costs. BOMP-NAS can find these neural networks at a 6x shorter search time compared to the closest related work.
  arXiv  Detail & Related papers  (2023-01-27T16:04:34Z)
• Sub 8-Bit Quantization of Streaming Keyword Spotting Models for Embedded Chipsets [7.5195830365852085]
  We propose a novel sub 8-bit quantization aware training algorithm for all components of a 250K parameter feedforward, streaming, state-free keyword spotting model. We conduct large scale experiments, training on 26,000 hours of de-identified production, far-field and near-field audio data.
  arXiv  Detail & Related papers  (2022-07-13T17:46:08Z)
• Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss [61.26793005355441]
  Cluster-Promoting Quantization (CPQ) finds the optimal quantization grids for neural networks. DropBits is a new bit-drop technique that revises the standard dropout regularization to randomly drop bits instead of neurons. We experimentally validate our method on various benchmark datasets and network architectures.
  arXiv  Detail & Related papers  (2021-09-05T15:15:07Z)
• Compact representations of convolutional neural networks via weight pruning and quantization [63.417651529192014]
  We propose a novel storage format for convolutional neural networks (CNNs) based on source coding and leveraging both weight pruning and quantization. We achieve a reduction of space occupancy up to 0.6% on fully connected layers and 5.44% on the whole network, while performing at least as competitive as the baseline.
  arXiv  Detail & Related papers  (2021-08-28T20:39:54Z)
• Differentiable Model Compression via Pseudo Quantization Noise [99.89011673907814]
  We propose to add independent pseudo quantization noise to model parameters during training to approximate the effect of a quantization operator. We experimentally verify that our method outperforms state-of-the-art quantization techniques on several benchmarks and architectures for image classification, language modeling, and audio source separation.
  arXiv  Detail & Related papers  (2021-04-20T14:14:03Z)
• Resource-efficient DNNs for Keyword Spotting using Neural Architecture Search and Quantization [23.850887499271842]
  This paper introduces neural architecture search (NAS) for the automatic discovery of small models for keyword spotting. We employ a differentiable NAS approach to optimize the structure of convolutional neural networks (CNNs) to maximize the classification accuracy. Using NAS only, we were able to obtain a highly efficient model with 95.4% accuracy on the Google speech commands dataset.
  arXiv  Detail & Related papers  (2020-12-18T09:53:55Z)
• Subtensor Quantization for Mobilenets [5.735035463793008]
  Quantization for deep neural networks (DNN) have enabled developers to deploy models with less memory and more efficient low-power inference. In this paper, we analyzed several root causes of quantization loss and proposed alternatives that do not rely on per-channel or training-aware approaches. We evaluate the image classification task on ImageNet dataset, and our post-training quantized 8-bit inference top-1 accuracy in within 0.7% of the floating point version.
  arXiv  Detail & Related papers  (2020-11-04T15:41:47Z)
• Learned Low Precision Graph Neural Networks [10.269500440688306]
  We show how to systematically quantise Deep Graph Neural Networks (GNNs) with minimal or no loss in performance using Network Architecture Search (NAS) The proposed novel NAS mechanism, named Low Precision Graph NAS (LPGNAS), constrains both architecture and quantisation choices to be differentiable. On eight different datasets, solving the task of classifying unseen nodes in a graph, LPGNAS generates quantised models with significant reductions in both model and buffer sizes.
  arXiv  Detail & Related papers  (2020-09-19T13:51:09Z)
• Leveraging Automated Mixed-Low-Precision Quantization for tiny edge microcontrollers [76.30674794049293]
  This paper presents an automated mixed-precision quantization flow based on the HAQ framework but tailored for the memory and computational characteristics of MCU devices. Specifically, a Reinforcement Learning agent searches for the best uniform quantization levels, among 2, 4, 8 bits, of individual weight and activation tensors. Given an MCU-class memory bound to 2MB for weight-only quantization, the compressed models produced by the mixed-precision engine result as accurate as the state-of-the-art solutions.
  arXiv  Detail & Related papers  (2020-08-12T06:09:58Z)
• Widening and Squeezing: Towards Accurate and Efficient QNNs [125.172220129257]
  Quantization neural networks (QNNs) are very attractive to the industry because their extremely cheap calculation and storage overhead, but their performance is still worse than that of networks with full-precision parameters. Most of existing methods aim to enhance performance of QNNs especially binary neural networks by exploiting more effective training techniques. We address this problem by projecting features in original full-precision networks to high-dimensional quantization features.
  arXiv  Detail & Related papers  (2020-02-03T04:11:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.