Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition

• URL: http://arxiv.org/abs/2102.07259v1
• Date: Sun, 14 Feb 2021 22:28:55 GMT
• Title: Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition
• Abstract summary: Attention is a very popular and effective mechanism in artificial neural network-based sequence-to-sequence models. Paper focuses on the development and evolution of attention models for offline and streaming speech recognition.
• Score: 4.542616945567623
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Attention is a very popular and effective mechanism in artificial neural network-based sequence-to-sequence models. In this survey paper, a comprehensive review of the different attention models used in developing automatic speech recognition systems is provided. The paper focuses on the development and evolution of attention models for offline and streaming speech recognition within recurrent neural network- and Transformer- based architectures.

Related papers

• Explainable AI in Speaker Recognition -- Attention Map Visualisation and Evaluation [21.594428525994285]
  This work aims to study the attention mechanisms of neural networks.<n>Experiments are performed on speaker recognition neural networks that are trained to identify speaker identity from a given utterance.
  arXiv  Detail & Related papers  (2026-06-22T06:31:26Z)
• Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks [59.38765771221084]
  We present a physiologically inspired speech recognition architecture compatible and scalable with deep learning frameworks. We show end-to-end gradient descent training leads to the emergence of neural oscillations in the central spiking neural network. Our findings highlight the crucial inhibitory role of feedback mechanisms, such as spike frequency adaptation and recurrent connections, in regulating and synchronising neural activity to improve recognition performance.
  arXiv  Detail & Related papers  (2024-04-22T09:40:07Z)
• Deep Neural Networks for Automatic Speaker Recognition Do Not Learn Supra-Segmental Temporal Features [2.724035499453558]
  We present and apply a novel test to quantify to what extent the performance of state-of-the-art neural networks for speaker recognition can be explained by modeling SST. We find that a variety of CNN- and RNN-based neural network architectures for speaker recognition do not model SST to any sufficient degree, even when forced.
  arXiv  Detail & Related papers  (2023-11-01T12:45:31Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Contrastive-Signal-Dependent Plasticity: Self-Supervised Learning in Spiking Neural Circuits [61.94533459151743]
  This work addresses the challenge of designing neurobiologically-motivated schemes for adjusting the synapses of spiking networks. Our experimental simulations demonstrate a consistent advantage over other biologically-plausible approaches when training recurrent spiking networks.
  arXiv  Detail & Related papers  (2023-03-30T02:40:28Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Full-Reference Speech Quality Estimation with Attentional Siamese Neural Networks [0.0]
  We present a full-reference speech quality prediction model with a deep learning approach. The model determines a feature representation of the reference and the degraded signal through a siamese recurrent convolutional network. The resulting features are then used to align the signals with an attention mechanism and are finally combined to estimate the overall speech quality.
  arXiv  Detail & Related papers  (2021-05-03T12:38:25Z)
• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview [43.12352697785169]
  We present a structured overview of adaptation algorithms for neural network-based speech recognition. The overview characterizes adaptation algorithms as based on embeddings, model parameter adaptation, or data augmentation. We present a meta-analysis of the performance of speech recognition adaptation algorithms, based on relative error rate reductions as reported in the literature.
  arXiv  Detail & Related papers  (2020-08-14T21:50:03Z)
• Visual Pattern Recognition with on On-chip Learning: towards a Fully Neuromorphic Approach [10.181725314550823]
  We present a spiking neural network (SNN) for visual pattern recognition with on-chip learning on neuromorphic hardware. We show how this network can learn simple visual patterns composed of horizontal and vertical bars sensed by a Dynamic Vision Sensor. During recognition, the network classifies the pattern's identity while at the same time estimating its location and scale.
  arXiv  Detail & Related papers  (2020-08-08T08:07:36Z)
• Multi-modal Automated Speech Scoring using Attention Fusion [46.94442359735952]
  We propose a novel multi-modal end-to-end neural approach for automated assessment of non-native English speakers' spontaneous speech. We employ Bi-directional Recurrent Convolutional Neural Networks and Bi-directional Long Short-Term Memory Neural Networks to encode acoustic and lexical cues from spectrograms and transcriptions. We find combined attention to both lexical and acoustic cues significantly improves the overall performance of the system.
  arXiv  Detail & Related papers  (2020-05-17T07:53:15Z)
• AutoSpeech: Neural Architecture Search for Speaker Recognition [108.69505815793028]
  We propose the first neural architecture search approach approach for the speaker recognition tasks, named as AutoSpeech. Our algorithm first identifies the optimal operation combination in a neural cell and then derives a CNN model by stacking the neural cell for multiple times. Results demonstrate that the derived CNN architectures significantly outperform current speaker recognition systems based on VGG-M, ResNet-18, and ResNet-34 back-bones, while enjoying lower model complexity.
  arXiv  Detail & Related papers  (2020-05-07T02:53:47Z)
• Intrinsic Motivation and Episodic Memories for Robot Exploration of High-Dimensional Sensory Spaces [0.0]
  This work presents an architecture that generates curiosity-driven goal-directed exploration behaviours for an image sensor of a microfarming robot. A combination of deep neural networks for offline unsupervised learning of low-dimensional features from images, and of online learning of shallow neural networks representing the inverse and forward kinematics of the system have been used. The artificial curiosity system assigns interest values to a set of pre-defined goals, and drives the exploration towards those that are expected to maximise the learning progress.
  arXiv  Detail & Related papers  (2020-01-07T11:39:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.