Object-aware Sound Source Localization via Audio-Visual Scene Understanding

• URL: http://arxiv.org/abs/2506.18557v2
• Date: Tue, 24 Jun 2025 02:05:51 GMT
• Title: Object-aware Sound Source Localization via Audio-Visual Scene Understanding
• Authors: Sung Jin Um, Dongjin Kim, Sangmin Lee, Jung Uk Kim,
• Abstract summary: Existing methods struggle with accurately localizing sound-making objects in complex scenes.<n>This limitation arises primarily from their reliance on simple audio-visual correspondence.<n>We propose a novel sound source localization framework leveraging Multimodal Large Language Models.
• Score: 14.801564966406486
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio-visual sound source localization task aims to spatially localize sound-making objects within visual scenes by integrating visual and audio cues. However, existing methods struggle with accurately localizing sound-making objects in complex scenes, particularly when visually similar silent objects coexist. This limitation arises primarily from their reliance on simple audio-visual correspondence, which does not capture fine-grained semantic differences between sound-making and silent objects. To address these challenges, we propose a novel sound source localization framework leveraging Multimodal Large Language Models (MLLMs) to generate detailed contextual information that explicitly distinguishes between sound-making foreground objects and silent background objects. To effectively integrate this detailed information, we introduce two novel loss functions: Object-aware Contrastive Alignment (OCA) loss and Object Region Isolation (ORI) loss. Extensive experimental results on MUSIC and VGGSound datasets demonstrate the effectiveness of our approach, significantly outperforming existing methods in both single-source and multi-source localization scenarios. Code and generated detailed contextual information are available at: https://github.com/VisualAIKHU/OA-SSL.

Related papers

• Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge [14.801564966406486]
  The goal of the multi-sound source localization task is to localize sound sources from the mixture individually. We present a novel multi-sound source localization method that can perform localization without prior knowledge of the number of sound sources.
  arXiv  Detail & Related papers  (2024-03-26T06:27:50Z)
• QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition [47.103732403296654]
  Multi-source semantic space can be represented as the Cartesian product of single-source sub-spaces. We introduce a global-to-local quantization mechanism, which distills knowledge from stable global (clip-level) features into local (frame-level) ones. Experiments demonstrate that our semantically decomposed audio representation significantly improves AVS performance.
  arXiv  Detail & Related papers  (2023-09-29T20:48:44Z)
• Sound Source Localization is All about Cross-Modal Alignment [53.957081836232206]
  Cross-modal semantic understanding is essential for genuine sound source localization. We propose a joint task with sound source localization to better learn the interaction between audio and visual modalities. Our method outperforms the state-of-the-art approaches in both sound source localization and cross-modal retrieval.
  arXiv  Detail & Related papers  (2023-09-19T16:04:50Z)
• Audio-Visual Spatial Integration and Recursive Attention for Robust Sound Source Localization [13.278494654137138]
  Humans utilize both audio and visual modalities as spatial cues to locate sound sources. We propose an audio-visual spatial integration network that integrates spatial cues from both modalities. Our method can perform more robust sound source localization.
  arXiv  Detail & Related papers  (2023-08-11T11:57:58Z)
• Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention [54.4258176885084]
  How to accurately recognize ambiguous sounds is a major challenge for audio captioning. We propose visually-aware audio captioning, which makes use of visual information to help the description of ambiguous sounding objects. Our proposed method achieves state-of-the-art results on machine translation metrics.
  arXiv  Detail & Related papers  (2022-10-28T22:45:41Z)
• Visual Sound Localization in the Wild by Cross-Modal Interference Erasing [90.21476231683008]
  In real-world scenarios, audios are usually contaminated by off-screen sound and background noise. We propose the Interference Eraser (IEr) framework, which tackles the problem of audio-visual sound source localization in the wild.
  arXiv  Detail & Related papers  (2022-02-13T21:06:19Z)
• Class-aware Sounding Objects Localization via Audiovisual Correspondence [51.39872698365446]
  We propose a two-stage step-by-step learning framework to localize and recognize sounding objects in complex audiovisual scenarios. We generate class-aware object localization maps in cocktail-party scenarios and use audiovisual correspondence to suppress silent areas. Experiments on both realistic and synthesized videos show that our model is superior in localizing and recognizing objects as well as filtering out silent ones.
  arXiv  Detail & Related papers  (2021-12-22T09:34:33Z)
• Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos [101.83513408195692]
  We propose a multitask learning method for visual-audio saliency prediction and sound source localization on multi-face video. The proposed method outperforms 12 state-of-the-art saliency prediction methods, and achieves competitive results in sound source localization.
  arXiv  Detail & Related papers  (2021-11-05T14:35:08Z)
• Dual Normalization Multitasking for Audio-Visual Sounding Object Localization [0.0]
  We propose a new concept, Sounding Object, to reduce the ambiguity of the visual location of sound. To tackle this new AVSOL problem, we propose a novel multitask training strategy and architecture called Dual Normalization Multitasking.
  arXiv  Detail & Related papers  (2021-06-01T02:02:52Z)
• Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching [87.42246194790467]
  We propose a two-stage learning framework to perform self-supervised class-aware sounding object localization. We show that our model is superior in filtering out silent objects and pointing out the location of sounding objects of different classes.
  arXiv  Detail & Related papers  (2020-10-12T05:51:55Z)
• Multiple Sound Sources Localization from Coarse to Fine [41.56420350529494]
  How to visually localize multiple sound sources in unconstrained videos is a formidable problem. We develop a two-stage audiovisual learning framework that disentangles audio and visual representations of different categories from complex scenes. Our model achieves state-of-the-art results on public dataset of localization.
  arXiv  Detail & Related papers  (2020-07-13T12:59:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.