Semantic Grouping Network for Audio Source Separation

• URL: http://arxiv.org/abs/2407.03736v1
• Date: Thu, 4 Jul 2024 08:37:47 GMT
• Title: Semantic Grouping Network for Audio Source Separation
• Authors: Shentong Mo, Yapeng Tian,
• Abstract summary: We present a novel Semantic Grouping Network, termed as SGN, that can directly disentangle sound representations and extract high-level semantic information for each source from input audio mixture. We conducted extensive experiments on music-only and universal sound separation benchmarks: MUSIC, FUSS, MUSDB18, and VGG-Sound.
• Score: 41.54814517077309
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, audio-visual separation approaches have taken advantage of the natural synchronization between the two modalities to boost audio source separation performance. They extracted high-level semantics from visual inputs as the guidance to help disentangle sound representation for individual sources. Can we directly learn to disentangle the individual semantics from the sound itself? The dilemma is that multiple sound sources are mixed together in the original space. To tackle the difficulty, in this paper, we present a novel Semantic Grouping Network, termed as SGN, that can directly disentangle sound representations and extract high-level semantic information for each source from input audio mixture. Specifically, SGN aggregates category-wise source features through learnable class tokens of sounds. Then, the aggregated semantic features can be used as the guidance to separate the corresponding audio sources from the mixture. We conducted extensive experiments on music-only and universal sound separation benchmarks: MUSIC, FUSS, MUSDB18, and VGG-Sound. The results demonstrate that our SGN significantly outperforms previous audio-only methods and audio-visual models without utilizing additional visual cues.

Related papers

• T-VSL: Text-Guided Visual Sound Source Localization in Mixtures [33.28678401737415]
  We develop a framework to disentangle audio-visual source correspondence from multi-source mixtures. Our framework exhibits promising zero-shot transferability to unseen classes during test time. Experiments conducted on the MUSIC, VGGSound, and VGGSound-Instruments datasets demonstrate significant performance improvements over state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-04-02T09:07:05Z)
• LAVSS: Location-Guided Audio-Visual Spatial Audio Separation [52.44052357829296]
  We propose a location-guided audio-visual spatial audio separator. The proposed LAVSS is inspired by the correlation between spatial audio and visual location. In addition, we adopt a pre-trained monaural separator to transfer knowledge from rich mono sounds to boost spatial audio separation.
  arXiv  Detail & Related papers  (2023-10-31T13:30:24Z)
• QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition [47.103732403296654]
  Multi-source semantic space can be represented as the Cartesian product of single-source sub-spaces. We introduce a global-to-local quantization mechanism, which distills knowledge from stable global (clip-level) features into local (frame-level) ones. Experiments demonstrate that our semantically decomposed audio representation significantly improves AVS performance.
  arXiv  Detail & Related papers  (2023-09-29T20:48:44Z)
• Separate Anything You Describe [55.0784713558149]
  Language-queried audio source separation (LASS) is a new paradigm for computational auditory scene analysis (CASA) AudioSep is a foundation model for open-domain audio source separation with natural language queries.
  arXiv  Detail & Related papers  (2023-08-09T16:09:44Z)
• Audio-Visual Grouping Network for Sound Localization from Mixtures [30.756247389435803]
  Previous single-source methods mainly used the audio-visual association as clues to localize sounding objects in each image. We propose a novel audio-visual grouping network, namely AVGN, that can directly learn category-wise semantic features for each source from the input audio mixture and image. Compared to existing multi-source methods, our new framework can localize a flexible number of sources and disentangle category-aware audio-visual representations for individual sound sources.
  arXiv  Detail & Related papers  (2023-03-29T22:58:55Z)
• Separate What You Describe: Language-Queried Audio Source Separation [53.65665794338574]
  We introduce the task of language-queried audio source separation (LASS) LASS aims to separate a target source from an audio mixture based on a natural language query of the target source. We propose LASS-Net, an end-to-end neural network that is learned to jointly process acoustic and linguistic information.
  arXiv  Detail & Related papers  (2022-03-28T23:47:57Z)
• Visual Scene Graphs for Audio Source Separation [65.47212419514761]
  State-of-the-art approaches for visually-guided audio source separation typically assume sources that have characteristic sounds, such as musical instruments. We propose Audio Visual Scene Graph Segmenter (AVSGS), a novel deep learning model that embeds the visual structure of the scene as a graph and segments this graph into subgraphs. Our pipeline is trained end-to-end via a self-supervised task consisting of separating audio sources using the visual graph from artificially mixed sounds.
  arXiv  Detail & Related papers  (2021-09-24T13:40:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.