Audio Mamba: Pretrained Audio State Space Model For Audio Tagging

• URL: http://arxiv.org/abs/2405.13636v1
• Date: Wed, 22 May 2024 13:35:56 GMT
• Title: Audio Mamba: Pretrained Audio State Space Model For Audio Tagging
• Authors: Jiaju Lin, Haoxuan Hu,
• Abstract summary: We propose Audio Mamba, a self-attention-free approach that captures long audio spectrogram dependency with state space models. Our experimental results on two audio-tagging datasets demonstrate the parameter efficiency of Audio Mamba, it achieves comparable results to SOTA audio spectrogram transformers with one third parameters.
• Score: 1.2123876307427102
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio tagging is an important task of mapping audio samples to their corresponding categories. Recently endeavours that exploit transformer models in this field have achieved great success. However, the quadratic self-attention cost limits the scaling of audio transformer models and further constrains the development of more universal audio models. In this paper, we attempt to solve this problem by proposing Audio Mamba, a self-attention-free approach that captures long audio spectrogram dependency with state space models. Our experimental results on two audio-tagging datasets demonstrate the parameter efficiency of Audio Mamba, it achieves comparable results to SOTA audio spectrogram transformers with one third parameters.

Related papers

• Beyond Single-Audio: Advancing Multi-Audio Processing in Audio Large Language Models [56.776580717999806]
  Real-world applications often involve processing multiple audio streams simultaneously. We propose the first multi-audio evaluation benchmark that consists of 20 datasets from 11 multi-audio tasks. We propose a novel multi-audio-LLM (MALLM) to capture audio context among multiple similar audios.
  arXiv  Detail & Related papers  (2024-09-27T12:06:53Z)
• Taming Data and Transformers for Audio Generation [49.54707963286065]
  AutoCap is a high-quality and efficient automatic audio captioning model. GenAu is a scalable transformer-based audio generation architecture. We compile 57M ambient audio clips, forming AutoReCap-XL, the largest available audio-text dataset.
  arXiv  Detail & Related papers  (2024-06-27T17:58:54Z)
• Audio Mamba: Bidirectional State Space Model for Audio Representation Learning [15.472819870523093]
  We introduce Audio Mamba, the first self-attention-free, purely SSM-based model for audio classification. We evaluate AuM on various audio datasets - comprising six different benchmarks - where it achieves comparable or better performance.
  arXiv  Detail & Related papers  (2024-06-05T15:00:59Z)
• Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations [16.269123889392343]
  This work proposes Audio Mamba, a selective state space model for learning general-purpose audio representations. Empirical results on ten diverse audio recognition downstream tasks show that the proposed models consistently outperform comparable self-supervised audio spectrogram transformer baselines.
  arXiv  Detail & Related papers  (2024-06-04T10:19:14Z)
• Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching [51.70360630470263]
  Video-to-audio (V2A) generation aims to synthesize content-matching audio from silent video. We propose Frieren, a V2A model based on rectified flow matching. Experiments indicate that Frieren achieves state-of-the-art performance in both generation quality and temporal alignment.
  arXiv  Detail & Related papers  (2024-06-01T06:40:22Z)
• AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes [6.375996974877916]
  We propose a method named AudioFormer, which learns audio feature representations through the acquisition of discrete acoustic codes. Our research outcomes demonstrate that AudioFormer attains significantly improved performance compared to prevailing monomodal audio classification models.
  arXiv  Detail & Related papers  (2023-08-14T15:47:25Z)
• Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models [65.18102159618631]
  multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. We propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps.
  arXiv  Detail & Related papers  (2023-01-30T04:44:34Z)
• AudioGen: Textually Guided Audio Generation [116.57006301417306]
  We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive model that generates audio samples conditioned on text inputs.
  arXiv  Detail & Related papers  (2022-09-30T10:17:05Z)
• BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis [129.86743102915986]
  We formulate the synthesis process from a different perspective by decomposing the audio into a common part. We propose BinauralGrad, a novel two-stage framework equipped with diffusion models to synthesize them respectively. Experiment results show that BinauralGrad outperforms the existing baselines by a large margin in terms of both object and subject evaluation metrics.
  arXiv  Detail & Related papers  (2022-05-30T02:09:26Z)
• Large Scale Audiovisual Learning of Sounds with Weakly Labeled Data [9.072124914105325]
  We present an audiovisual fusion model that learns to recognize sounds from weakly labeled video recordings. Experiments on the large scale sound events dataset, AudioSet, demonstrate the efficacy of the proposed model.
  arXiv  Detail & Related papers  (2020-05-29T01:30:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.