State-Space Large Audio Language Models

• URL: http://arxiv.org/abs/2411.15685v1
• Date: Sun, 24 Nov 2024 02:21:28 GMT
• Title: State-Space Large Audio Language Models
• Authors: Saurabhchand Bhati, Yuan Gong, Leonid Karlinsky, Hilde Kuehne, Rogerio Feris, James Glass,
• Abstract summary: Large Audio Language Models (LALM) combine the audio perception models and the Large Language Models (LLM) These systems rely on Transformers which scale quadratically with the input sequence lengths which poses computational challenges in deploying these systems in memory and time-constrained scenarios. We propose the first state-space-based LALM. Experimental results demonstrate that space-based LALM despite having a significantly lower number of parameters performs competitively with transformer-based LALMs on close-ended tasks on a variety of datasets.
• Score: 38.65927845229572
• License:
• Abstract: Large Audio Language Models (LALM) combine the audio perception models and the Large Language Models (LLM) and show a remarkable ability to reason about the input audio, infer the meaning, and understand the intent. However, these systems rely on Transformers which scale quadratically with the input sequence lengths which poses computational challenges in deploying these systems in memory and time-constrained scenarios. Recently, the state-space models (SSMs) have emerged as an alternative to transformer networks. While there have been successful attempts to replace transformer-based audio perception models with state-space ones, state-space-based LALMs remain unexplored. First, we begin by replacing the transformer-based audio perception module and then replace the transformer-based LLM and propose the first state-space-based LALM. Experimental results demonstrate that space-based LALM despite having a significantly lower number of parameters performs competitively with transformer-based LALMs on close-ended tasks on a variety of datasets.

Related papers

• SLMGAN: Exploiting Speech Language Model Representations for Unsupervised Zero-Shot Voice Conversion in GANs [22.522376665078248]
  This paper introduces a new approach, SLMGAN, to leverage SLM representations for discriminative tasks within the generative adversarial network (GAN) framework. Building upon StarGANv2-VC, we add our novel SLM-based WavLM discriminators on top of the mel-based discriminators along with our newly designed SLM feature matching loss function. Subjective evaluation results show that SLMGAN outperforms existing state-of-the-art zero-shot voice conversion models in terms of naturalness and achieves comparable similarity.
  arXiv  Detail & Related papers  (2023-07-18T17:09:15Z)
• Multi-Head State Space Model for Speech Recognition [44.04124537862432]
  State space models (SSMs) have recently shown promising results on small-scale sequence and language modelling tasks. In this paper, we propose a multi-head state space (MH-SSM) architecture equipped with special gating mechanisms. As a drop-in replacement for multi-head attention in transformer encoders, this new model significantly outperforms the transformer transducer on the LibriSpeech speech recognition corpus.
  arXiv  Detail & Related papers  (2023-05-21T16:28:57Z)
• MAST: Multiscale Audio Spectrogram Transformers [53.06337011259031]
  We present Multiscale Audio Spectrogram Transformer (MAST) for audio classification, which brings the concept of multiscale feature hierarchies to the Audio Spectrogram Transformer (AST) In practice, MAST significantly outperforms AST by an average accuracy of 3.4% across 8 speech and non-speech tasks from the LAPE Benchmark.
  arXiv  Detail & Related papers  (2022-11-02T23:34:12Z)
• Learning Bounded Context-Free-Grammar via LSTM and the Transformer:Difference and Explanations [51.77000472945441]
  Long Short-Term Memory (LSTM) and Transformers are two popular neural architectures used for natural language processing tasks. In practice, it is often observed that Transformer models have better representation power than LSTM. We study such practical differences between LSTM and Transformer and propose an explanation based on their latent space decomposition patterns.
  arXiv  Detail & Related papers  (2021-12-16T19:56:44Z)
• Bayesian Transformer Language Models for Speech Recognition [59.235405107295655]
  State-of-the-art neural language models (LMs) represented by Transformers are highly complex. This paper proposes a full Bayesian learning framework for Transformer LM estimation.
  arXiv  Detail & Related papers  (2021-02-09T10:55:27Z)
• Wake Word Detection with Streaming Transformers [72.66551640048405]
  We show that our proposed Transformer model outperforms the baseline convolution network by 25% on average in false rejection rate at the same false alarm rate. Our experiments on the Mobvoi wake word dataset demonstrate that our proposed Transformer model outperforms the baseline convolution network by 25%.
  arXiv  Detail & Related papers  (2021-02-08T19:14:32Z)
• Parameter Efficient Multimodal Transformers for Video Representation Learning [108.8517364784009]
  This work focuses on reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning. We show that our approach reduces parameters up to 80$%$, allowing us to train our model end-to-end from scratch. To demonstrate our approach, we pretrain our model on 30-second clips from Kinetics-700 and transfer it to audio-visual classification tasks.
  arXiv  Detail & Related papers  (2020-12-08T00:16:13Z)
• Transformer in action: a comparative study of transformer-based acoustic models for large scale speech recognition applications [23.470690511056173]
  We compare transformer based acoustic models with their LSTM counterparts on industrial scale tasks. On a low latency voice assistant task, Emformer gets 24% to 26% relative word error rate reductions (WERRs) For medium latency scenarios, comparing with LCBLSTM with similar model size and latency, Emformer gets significant WERR across four languages in video captioning datasets.
  arXiv  Detail & Related papers  (2020-10-27T23:04:21Z)
• LSTM-LM with Long-Term History for First-Pass Decoding in Conversational Speech Recognition [27.639919625398]
  LSTM language models (LSTM-LMs) have been proven to be powerful and yielded significant performance improvements over count based n-gram LMs in modern speech recognition systems. Recent work shows that it is feasible and computationally affordable to adopt the LSTM-LMs in the first-pass decoding within a dynamic (or tree based) decoder framework.
  arXiv  Detail & Related papers  (2020-10-21T23:40:26Z)
• Exploring Transformers for Large-Scale Speech Recognition [34.645597506707055]
  We show that Transformers can achieve around 6% relative word error rate (WER) reduction compared to the BLSTM baseline in the offline fashion. In the streaming fashion, Transformer-XL is comparable to LC-BLSTM with 800 millisecond latency constraint.
  arXiv  Detail & Related papers  (2020-05-19T18:07:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.