Weakly-supervised Audio Separation via Bi-modal Semantic Similarity

• URL: http://arxiv.org/abs/2404.01740v1
• Date: Tue, 2 Apr 2024 08:59:58 GMT
• Title: Weakly-supervised Audio Separation via Bi-modal Semantic Similarity
• Authors: Tanvir Mahmud, Saeed Amizadeh, Kazuhito Koishida, Diana Marculescu,
• Abstract summary: Existing mix-and-separate based methods suffer from significant performance drop with multi-source training mixtures. We propose a generic bi-modal separation framework which can enhance the existing unsupervised frameworks to separate single-source signals in a target modality. We show that our framework can achieve 71% boost in terms of Signal-to-Distortion Ratio (SDR) over the baseline, reaching 97.5% of the supervised learning performance.
• Score: 21.610354683236885
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conditional sound separation in multi-source audio mixtures without having access to single source sound data during training is a long standing challenge. Existing mix-and-separate based methods suffer from significant performance drop with multi-source training mixtures due to the lack of supervision signal for single source separation cases during training. However, in the case of language-conditional audio separation, we do have access to corresponding text descriptions for each audio mixture in our training data, which can be seen as (rough) representations of the audio samples in the language modality. To this end, in this paper, we propose a generic bi-modal separation framework which can enhance the existing unsupervised frameworks to separate single-source signals in a target modality (i.e., audio) using the easily separable corresponding signals in the conditioning modality (i.e., language), without having access to single-source samples in the target modality during training. We empirically show that this is well within reach if we have access to a pretrained joint embedding model between the two modalities (i.e., CLAP). Furthermore, we propose to incorporate our framework into two fundamental scenarios to enhance separation performance. First, we show that our proposed methodology significantly improves the performance of purely unsupervised baselines by reducing the distribution shift between training and test samples. In particular, we show that our framework can achieve 71% boost in terms of Signal-to-Distortion Ratio (SDR) over the baseline, reaching 97.5% of the supervised learning performance. Second, we show that we can further improve the performance of the supervised learning itself by 17% if we augment it by our proposed weakly-supervised framework, that enables a powerful semi-supervised framework for audio separation.

Related papers

• SSAVSV: Towards Unified Model for Self-Supervised Audio-Visual Speaker Verification [3.380873355096444]
  We propose a self-supervised learning framework based on contrastive learning with asymmetric masking and masked data modeling.<n>We employ a unified framework for self-supervised audiovisual speaker verification using a single shared backbone for audio and visual inputs.<n>Our method achieves competitive performance without labeled data while reducing computational costs compared to traditional approaches.
  arXiv  Detail & Related papers  (2025-06-21T12:02:53Z)
• Training-Free Multi-Step Audio Source Separation [16.187944473839632]
  We show that pretrained one-step audio source separation models can be leveraged for multi-step separation without additional training.<n>We propose a simple yet effective inference method that iteratively applies separation by optimally blending the input mixture with the previous step's separation result.<n>Our empirical results demonstrate that our multi-step separation approach consistently outperforms one-step inference across both speech enhancement and music source separation tasks.
  arXiv  Detail & Related papers  (2025-05-26T05:40:12Z)
• A contrastive-learning approach for auditory attention detection [11.28441753596964]
  We propose a method based on self supervised learning to minimize the difference between the latent representations of an attended speech signal and the corresponding EEG signal. We compare our results with previously published methods and achieve state-of-the-art performance on the validation set.
  arXiv  Detail & Related papers  (2024-10-24T03:13:53Z)
• CPM: Class-conditional Prompting Machine for Audio-visual Segmentation [17.477225065057993]
  Class-conditional Prompting Machine (CPM) improves bipartite matching with a learning strategy combining class-agnostic queries with class-conditional queries. We conduct experiments on AVS benchmarks, demonstrating that our method achieves state-of-the-art (SOTA) segmentation accuracy.
  arXiv  Detail & Related papers  (2024-07-07T13:20:21Z)
• Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder [58.523884148942166]
  We propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword units in Mandarin to establish good frame-level syllable boundaries from lip shapes. Next, we propose an audio-guided cross-modal fusion encoder (CMFE) neural network to utilize main training parameters for multiple cross-modal attention layers.
  arXiv  Detail & Related papers  (2023-08-14T08:19:24Z)
• Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification [55.624946113550195]
  This paper proposes a cross-modal speech co-learning paradigm. Two cross-modal boosters are introduced based on an audio-visual pseudo-siamese structure to learn the modality-transformed correlation. Experimental results on the LRSLip3, GridLip, LomGridLip, and VoxLip datasets demonstrate that our proposed method achieves 60% and 20% average relative performance improvement.
  arXiv  Detail & Related papers  (2023-02-22T10:06:37Z)
• Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing [52.2231419645482]
  This paper focuses on the weakly-supervised audio-visual video parsing task. It aims to recognize all events belonging to each modality and localize their temporal boundaries.
  arXiv  Detail & Related papers  (2022-04-25T11:41:17Z)
• Unsupervised Sound Localization via Iterative Contrastive Learning [106.56167882750792]
  We propose an iterative contrastive learning framework that requires no data annotations. We then use the pseudo-labels to learn the correlation between the visual and audio signals sampled from the same video. Our iterative strategy gradually encourages the localization of the sounding objects and reduces the correlation between the non-sounding regions and the reference audio.
  arXiv  Detail & Related papers  (2021-04-01T07:48:29Z)
• Robust Audio-Visual Instance Discrimination [79.74625434659443]
  We present a self-supervised learning method to learn audio and video representations. We address the problems of audio-visual instance discrimination and improve transfer learning performance.
  arXiv  Detail & Related papers  (2021-03-29T19:52:29Z)
• Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning [58.14807331265752]
  We show that better speaker embeddings can be learned by momentum contrastive learning. We generalize the self-supervised framework to a semi-supervised scenario where only a small portion of the data is labeled.
  arXiv  Detail & Related papers  (2020-12-13T23:23:39Z)
• Multimodal Semi-supervised Learning Framework for Punctuation Prediction in Conversational Speech [17.602098162338137]
  We explore a multimodal semi-supervised learning approach for punctuation prediction. We learn representations from large amounts of unlabelled audio and text data. When trained on 1 hour of speech and text data, the proposed model achieved 9-18% absolute improvement over baseline model.
  arXiv  Detail & Related papers  (2020-08-03T08:13:09Z)
• Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision [44.88044155505332]
  We build on earlier work to train embeddings that are more discriminative for uni-modal downstream tasks. We propose a novel training strategy that not only optimises metrics across modalities, but also enforces intra-class feature separation within each of the modalities. The effectiveness of the method is demonstrated on two downstream tasks: lip reading using the features trained on audio-visual synchronisation, and speaker recognition using the features trained for cross-modal biometric matching.
  arXiv  Detail & Related papers  (2020-04-29T16:51:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.