Resource-Efficient Speech Quality Prediction through Quantization Aware Training and Binary Activation Maps

• URL: http://arxiv.org/abs/2407.04578v1
• Date: Fri, 5 Jul 2024 15:15:00 GMT
• Title: Resource-Efficient Speech Quality Prediction through Quantization Aware Training and Binary Activation Maps
• Authors: Mattias Nilsson, Riccardo Miccini, Clément Laroche, Tobias Piechowiak, Friedemann Zenke,
• Abstract summary: We investigate binary activation maps (BAMs) for speech quality prediction on a convolutional architecture based on DNSMOS. We show that the binary activation model with quantization aware training matches the predictive performance of the baseline model. Our approach results in a 25-fold memory reduction during inference, while replacing almost all dot products with summations.
• Score: 4.002057316863807
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As speech processing systems in mobile and edge devices become more commonplace, the demand for unintrusive speech quality monitoring increases. Deep learning methods provide high-quality estimates of objective and subjective speech quality metrics. However, their significant computational requirements are often prohibitive on resource-constrained devices. To address this issue, we investigated binary activation maps (BAMs) for speech quality prediction on a convolutional architecture based on DNSMOS. We show that the binary activation model with quantization aware training matches the predictive performance of the baseline model. It further allows using other compression techniques. Combined with 8-bit weight quantization, our approach results in a 25-fold memory reduction during inference, while replacing almost all dot products with summations. Our findings show a path toward substantial resource savings by supporting mixed-precision binary multiplication in hard- and software.

Related papers

• Self-Supervised Speech Quality Estimation and Enhancement Using Only Clean Speech [50.95292368372455]
  We propose VQScore, a self-supervised metric for evaluating speech based on the quantization error of a vector-quantized-variational autoencoder (VQ-VAE) The training of VQ-VAE relies on clean speech; hence, large quantization errors can be expected when the speech is distorted. We found that the vector quantization mechanism could also be used for self-supervised speech enhancement (SE) model training.
  arXiv  Detail & Related papers  (2024-02-26T06:01:38Z)
• Enabling On-device Continual Learning with Binary Neural Networks [3.180732240499359]
  We propose a solution that combines recent advancements in the field of Continual Learning (CL) and Binary Neural Networks (BNNs) Specifically, our approach leverages binary latent replay activations and a novel quantization scheme that significantly reduces the number of bits required for gradient computation.
  arXiv  Detail & Related papers  (2024-01-18T11:57:05Z)
• Multimodal deep representation learning for quantum cross-platform verification [60.01590250213637]
  Cross-platform verification, a critical undertaking in the realm of early-stage quantum computing, endeavors to characterize the similarity of two imperfect quantum devices executing identical algorithms. We introduce an innovative multimodal learning approach, recognizing that the formalism of data in this task embodies two distinct modalities. We devise a multimodal neural network to independently extract knowledge from these modalities, followed by a fusion operation to create a comprehensive data representation.
  arXiv  Detail & Related papers  (2023-11-07T04:35:03Z)
• Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder [58.523884148942166]
  We propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword units in Mandarin to establish good frame-level syllable boundaries from lip shapes. Next, we propose an audio-guided cross-modal fusion encoder (CMFE) neural network to utilize main training parameters for multiple cross-modal attention layers.
  arXiv  Detail & Related papers  (2023-08-14T08:19:24Z)
• RAND: Robustness Aware Norm Decay For Quantized Seq2seq Models [14.07649230604283]
  We propose low complexity changes to the quantization aware training (QAT) process to improve model accuracy. With the improved accuracy, it opens up the possibility to exploit some of the other benefits of noise based QAT.
  arXiv  Detail & Related papers  (2023-05-24T19:45:56Z)
• Audio-Visual Efficient Conformer for Robust Speech Recognition [91.3755431537592]
  We propose to improve the noise of the recently proposed Efficient Conformer Connectionist Temporal Classification architecture by processing both audio and visual modalities. Our experiments show that using audio and visual modalities allows to better recognize speech in the presence of environmental noise and significantly accelerate training, reaching lower WER with 4 times less training steps.
  arXiv  Detail & Related papers  (2023-01-04T05:36:56Z)
• High Fidelity Neural Audio Compression [92.4812002532009]
  We introduce a state-of-the-art real-time, high-fidelity, audio leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary.
  arXiv  Detail & Related papers  (2022-10-24T17:52:02Z)
• Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer [1.9659095632676098]
  Quantizing weights and activations of deep neural networks is essential for deploying them in resource-constrained devices or cloud platforms. While binarization is a special case of quantization, this extreme case often leads to several training difficulties. We develop a unified quantization framework, denoted as UniQ, to overcome binarization difficulties.
  arXiv  Detail & Related papers  (2021-04-01T02:33:31Z)
• Streaming Attention-Based Models with Augmented Memory for End-to-End Speech Recognition [26.530909772863417]
  We build a compact and streaming speech recognition system on top of the end-to-end neural transducer architecture with attention-based modules augmented with convolution. The proposed system equips the end-to-end models with the streaming capability and reduces the large footprint from the streaming attention-based model using augmented memory. On the LibriSpeech dataset, our proposed system achieves word error rates 2.7% on test-clean and 5.8% on test-other, to our best knowledge the lowest among streaming approaches reported so far.
  arXiv  Detail & Related papers  (2020-11-03T00:43:58Z)
• Resource-Efficient Speech Mask Estimation for Multi-Channel Speech Enhancement [15.361841669377776]
  We provide a resource-efficient approach for multi-channel speech enhancement based on Deep Neural Networks (DNNs) In particular, we use reduced-precision DNNs for estimating a speech mask from noisy, multi-channel microphone observations. In the extreme case of binary weights and reduced precision activations, a significant reduction of execution time and memory footprint is possible.
  arXiv  Detail & Related papers  (2020-07-22T14:58:29Z)
• Deep Speaker Embeddings for Far-Field Speaker Recognition on Short Utterances [53.063441357826484]
  Speaker recognition systems based on deep speaker embeddings have achieved significant performance in controlled conditions. Speaker verification on short utterances in uncontrolled noisy environment conditions is one of the most challenging and highly demanded tasks. This paper presents approaches aimed to achieve two goals: a) improve the quality of far-field speaker verification systems in the presence of environmental noise, reverberation and b) reduce the system qualitydegradation for short utterances.
  arXiv  Detail & Related papers  (2020-02-14T13:34:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.