Skipformer: A Skip-and-Recover Strategy for Efficient Speech Recognition

• URL: http://arxiv.org/abs/2403.08258v2
• Date: Tue, 21 May 2024 02:06:26 GMT
• Title: Skipformer: A Skip-and-Recover Strategy for Efficient Speech Recognition
• Authors: Wenjing Zhu, Sining Sun, Changhao Shan, Peng Fan, Qing Yang,
• Abstract summary: Conformer-based attention models have become the de facto backbone model for Automatic Speech Recognition tasks. We propose a "Skip-and-Recover" Conformer architecture, named Skipformer, to squeeze sequence input length dynamically and inhomogeneously. Our model reduces the input sequence length by 31 times on Aishell-1 and 22 times on Librispeech corpus.
• Score: 7.963605445905696
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conformer-based attention models have become the de facto backbone model for Automatic Speech Recognition tasks. A blank symbol is usually introduced to align the input and output sequences for CTC or RNN-T models. Unfortunately, the long input length overloads computational budget and memory consumption quadratically by attention mechanism. In this work, we propose a "Skip-and-Recover" Conformer architecture, named Skipformer, to squeeze sequence input length dynamically and inhomogeneously. Skipformer uses an intermediate CTC output as criteria to split frames into three groups: crucial, skipping and ignoring. The crucial group feeds into next conformer blocks and its output joint with skipping group by original temporal order as the final encoder output. Experiments show that our model reduces the input sequence length by 31 times on Aishell-1 and 22 times on Librispeech corpus. Meanwhile, the model can achieve better recognition accuracy and faster inference speed than recent baseline models. Our code is open-sourced and available online.

Related papers

• Squeezed Attention: Accelerating Long Context Length LLM Inference [64.11145320159126]
  We propose Squeezed Attention as a mechanism to accelerate LLM applications where a large portion of the input prompt is fixed. We use K-means clustering offline to group the keys for the fixed context based on semantic similarity and represent each cluster with a single centroid value. We then compute exact attention using only these important keys from the fixed context, thereby reducing bandwidth and computational costs.
  arXiv  Detail & Related papers  (2024-11-14T18:54:19Z)
• Key Frame Mechanism For Efficient Conformer Based End-to-end Speech Recognition [9.803556181225193]
  Conformer as a backbone network for end-to-end automatic speech recognition achieved state-of-the-art performance. However, the Conformer-based model encounters an issue with the self-attention mechanism. We introduce key frame-based self-attention (KFSA) mechanism, a novel method to reduce the computation of the self-attention mechanism using key frames.
  arXiv  Detail & Related papers  (2023-10-23T13:55:49Z)
• Toeplitz Neural Network for Sequence Modeling [46.04964190407727]
  We show that a Toeplitz matrix-vector production trick can reduce the space-time complexity of the sequence modeling to log linear. A lightweight sub-network called relative position encoder is proposed to generate relative position coefficients with a fixed budget of parameters. Despite being trained on 512-token sequences, our model can extrapolate input sequence length up to 14K tokens in inference with consistent performance.
  arXiv  Detail & Related papers  (2023-05-08T14:49:01Z)
• When Counting Meets HMER: Counting-Aware Network for Handwritten Mathematical Expression Recognition [57.51793420986745]
  We propose an unconventional network for handwritten mathematical expression recognition (HMER) named Counting-Aware Network (CAN) We design a weakly-supervised counting module that can predict the number of each symbol class without the symbol-level position annotations. Experiments on the benchmark datasets for HMER validate that both joint optimization and counting results are beneficial for correcting the prediction errors of encoder-decoder models.
  arXiv  Detail & Related papers  (2022-07-23T08:39:32Z)
• Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences [52.6022911513076]
  Transformer-based models are not efficient in processing long sequences due to the quadratic space and time complexity of the self-attention modules. We propose Linformer and Informer to reduce the quadratic complexity to linear (modulo logarithmic factors) via low-dimensional projection and row selection. Based on the theoretical analysis, we propose Skeinformer to accelerate self-attention and further improve the accuracy of matrix approximation to self-attention.
  arXiv  Detail & Related papers  (2021-12-10T06:58:05Z)
• Rescoring Sequence-to-Sequence Models for Text Line Recognition with CTC-Prefixes [0.0]
  We propose to use the CTC-Prefix-Score during S2S decoding. During beam search, paths that are invalid according to the CTC confidence matrix are penalised. We evaluate this setup on three HTR data sets: IAM, Rimes, and StAZH.
  arXiv  Detail & Related papers  (2021-10-12T11:40:05Z)
• Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment [18.487842656780728]
  Infilling and iterative refinement models make up some of this gap by editing the outputs of a non-autoregressive model. We propose iterative realignment, where refinements occur over latent alignments rather than output sequence space.
  arXiv  Detail & Related papers  (2020-10-24T09:35:37Z)
• FastLR: Non-Autoregressive Lipreading Model with Integrate-and-Fire [74.04394069262108]
  We propose FastLR, a non-autoregressive (NAR) lipreading model which generates all target tokens simultaneously. FastLR achieves the speedup up to 10.97$times$ compared with state-of-the-art lipreading model.
  arXiv  Detail & Related papers  (2020-08-06T08:28:56Z)
• SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection [51.376723069962]
  We present a method for accelerating and structuring self-attentions: Sparse Adaptive Connection. In SAC, we regard the input sequence as a graph and attention operations are performed between linked nodes. We show that SAC is competitive with state-of-the-art models while significantly reducing memory cost.
  arXiv  Detail & Related papers  (2020-03-22T07:58:44Z)
• Imputer: Sequence Modelling via Imputation and Dynamic Programming [101.5705527605346]
  Imputer is an iterative generative model, requiring only a constant number of generation steps independent of the number of input or output tokens. We present a tractable dynamic programming training algorithm, which yields a lower bound on the log marginal likelihood.
  arXiv  Detail & Related papers  (2020-02-20T18:21:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.