Multi-View Spectrogram Transformer for Respiratory Sound Classification

• URL: http://arxiv.org/abs/2311.09655v3
• Date: Thu, 30 May 2024 05:42:03 GMT
• Title: Multi-View Spectrogram Transformer for Respiratory Sound Classification
• Authors: Wentao He, Yuchen Yan, Jianfeng Ren, Ruibin Bai, Xudong Jiang,
• Abstract summary: A Multi-View Spectrogram Transformer (MVST) is proposed to embed different views of time-frequency characteristics into the vision transformer. Experimental results on the ICBHI dataset demonstrate that the proposed MVST significantly outperforms state-of-the-art methods for classifying respiratory sounds.
• Score: 32.346046623638394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks have been applied to audio spectrograms for respiratory sound classification. Existing models often treat the spectrogram as a synthetic image while overlooking its physical characteristics. In this paper, a Multi-View Spectrogram Transformer (MVST) is proposed to embed different views of time-frequency characteristics into the vision transformer. Specifically, the proposed MVST splits the mel-spectrogram into different sized patches, representing the multi-view acoustic elements of a respiratory sound. These patches and positional embeddings are then fed into transformer encoders to extract the attentional information among patches through a self-attention mechanism. Finally, a gated fusion scheme is designed to automatically weigh the multi-view features to highlight the best one in a specific scenario. Experimental results on the ICBHI dataset demonstrate that the proposed MVST significantly outperforms state-of-the-art methods for classifying respiratory sounds.

Related papers

• Spectral and Rhythm Features for Audio Classification with Deep Convolutional Neural Networks [0.0]
  Convolutional neural networks (CNNs) are widely used in computer vision. They can be used to represent spectral and rhythm features extracted from digital imagery for the acoustic classification of sounds. Different spectral and rhythm feature representations like mel-scaled spectrograms, mel-frequency cepstral coefficients (MFCCs) are investigated.
  arXiv  Detail & Related papers  (2024-10-09T14:21:59Z)
• Convolutional Variational Autoencoders for Spectrogram Compression in Automatic Speech Recognition [0.0]
  This paper presents an alternative approach towards generating compressed spectrogram representation, based on Convolutional Variational Autoencoders (VAE) A Convolutional VAE model was trained on a subsample of the LibriSpeech dataset to reconstruct short fragments of audio spectrograms (25 ms) from a 13-dimensional embedding. The trained model for a 40-dimensional (300 ms) embedding was used to generate features for corpus of spoken commands on the GoogleSpeechCommands dataset.
  arXiv  Detail & Related papers  (2024-10-03T15:04:27Z)
• Dynamic Spectrum Mixer for Visual Recognition [17.180863898764194]
  We propose a content-adaptive yet computationally efficient structure, dubbed Dynamic Spectrum Mixer (DSM) DSM represents token interactions in the frequency domain by employing the Cosine Transform. It can learn long-term spatial dependencies with log-linear complexity.
  arXiv  Detail & Related papers  (2023-09-13T04:51:15Z)
• ViTs for SITS: Vision Transformers for Satellite Image Time Series [52.012084080257544]
  We introduce a fully-attentional model for general Satellite Image Time Series (SITS) processing based on the Vision Transformer (ViT) TSViT splits a SITS record into non-overlapping patches in space and time which are tokenized and subsequently processed by a factorized temporo-spatial encoder.
  arXiv  Detail & Related papers  (2023-01-12T11:33:07Z)
• Masked Frequency Modeling for Self-Supervised Visual Pre-Training [102.89756957704138]
  We present Masked Frequency Modeling (MFM), a unified frequency-domain-based approach for self-supervised pre-training of visual models. MFM first masks out a portion of frequency components of the input image and then predicts the missing frequencies on the frequency spectrum. For the first time, MFM demonstrates that, for both ViT and CNN, a simple non-Siamese framework can learn meaningful representations even using none of the following: (i) extra data, (ii) extra model, (iii) mask token.
  arXiv  Detail & Related papers  (2022-06-15T17:58:30Z)
• MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction [148.26195175240923]
  We propose a novel Transformer-based method, Multi-stage Spectral-wise Transformer (MST++) for efficient spectral reconstruction. In the NTIRE 2022 Spectral Reconstruction Challenge, our approach won the First place.
  arXiv  Detail & Related papers  (2022-04-17T02:39:32Z)
• Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction [127.20208645280438]
  Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-spectral signal from a 2D measurement. Modeling the inter-spectra interactions is beneficial for HSI reconstruction. Mask-guided Spectral-wise Transformer (MST) proposes a novel framework for HSI reconstruction.
  arXiv  Detail & Related papers  (2021-11-15T16:59:48Z)
• SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers [91.09957836250209]
  Hyperspectral (HS) images are characterized by approximately contiguous spectral information. CNNs have been proven to be a powerful feature extractor in HS image classification. We propose a novel backbone network called ulSpectralFormer for HS image classification.
  arXiv  Detail & Related papers  (2021-07-07T02:59:21Z)
• Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling [61.351967629600594]
  This paper proposes an any-to-many location-relative, sequence-to-sequence (seq2seq), non-parallel voice conversion approach. In this approach, we combine a bottle-neck feature extractor (BNE) with a seq2seq synthesis module. Objective and subjective evaluations show that the proposed any-to-many approach has superior voice conversion performance in terms of both naturalness and speaker similarity.
  arXiv  Detail & Related papers  (2020-09-06T13:01:06Z)
• CNN-MoE based framework for classification of respiratory anomalies and lung disease detection [33.45087488971683]
  This paper presents and explores a robust deep learning framework for auscultation analysis. It aims to classify anomalies in respiratory cycles and detect disease, from respiratory sound recordings.
  arXiv  Detail & Related papers  (2020-04-04T21:45:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.