Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning

• URL: http://arxiv.org/abs/2604.06824v1
• Date: Wed, 08 Apr 2026 08:40:33 GMT
• Title: Generate, Analyze, and Refine: Training-Free Sound Source Localization via MLLM Meta-Reasoning
• Abstract summary: Sound source localization task aims to identify the locations of sound-emitting objects by leveraging correlations between audio and visual modalities.<n>Most existing SSL methods rely on contrastive learning-based feature matching, but lack explicit reasoning and verification.<n>We propose a training-free SSL framework that exploits the intrinsic reasoning capabilities of Multimodal Large Language Models.
• Score: 24.266987119038134
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sound source localization task aims to identify the locations of sound-emitting objects by leveraging correlations between audio and visual modalities. Most existing SSL methods rely on contrastive learning-based feature matching, but lack explicit reasoning and verification, limiting their effectiveness in complex acoustic scenes. Inspired by human meta-cognitive processes, we propose a training-free SSL framework that exploits the intrinsic reasoning capabilities of Multimodal Large Language Models (MLLMs). Our Generation-Analysis-Refinement (GAR) pipeline consists of three stages: Generation produces initial bounding boxes and audio classifications; Analysis quantifies Audio-Visual Consistency via open-set role tagging and anchor voting; and Refinement applies adaptive gating to prevent unnecessary adjustments. Extensive experiments on single-source and multi-source benchmarks demonstrate competitive performance. The source code is available at https://github.com/VisualAIKHU/GAR-SSL.

Related papers

• Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources [57.68713138159972]
  We present a time-resolved audio encoder that learns event semantics together with source trajectories.<n>Experiments show that this representation improves the semanticlocalization and yields stronger reasoning than spatial and localization-oriented tradeoffs.
  arXiv  Detail & Related papers  (2026-06-12T05:58:31Z)
• JASTIN: Aligning LLMs for Zero-Shot Audio and Speech Evaluation via Natural Language Instructions [66.71387365429062]
  JASTIN is an instruction-driven audio evaluation framework.<n>It formulates audio assessment as a self-instructed reasoning task.<n>It consistently outperforms general MLLMs across speech, sound, music, and out-of-domain evaluation tasks.
  arXiv  Detail & Related papers  (2026-05-06T05:18:42Z)
• SICL-AT: Another way to adapt Auditory LLM to low-resource task [34.82834349882226]
  Auditory Large Language Models (LLMs) have demonstrated strong performance across a wide range of speech and audio understanding tasks.<n>They often struggle when applied to low-resource or unfamiliar tasks.<n>In-Context Learning (ICL) provides a training-free, inference-time solution.
  arXiv  Detail & Related papers  (2026-01-26T19:15:16Z)
• Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models [34.15708407614003]
  Large language models (LLMs) have recently achieved impressive results in speech recognition across multiple modalities.<n>We present Omni-AVSR, a unified audio-visual LLM that combines efficient multi-granularity training with parameter-efficient adaptation.<n> Experiments on LRS2 and LRS3 show that Omni-AVSR achieves comparable or superior accuracy to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-11-10T16:03:44Z)
• An Evaluation of Interleaved Instruction Tuning on Semantic Reasoning Performance in an Audio MLLM [15.340075567628466]
  This work examined the impact of interleaved instruction tuning in an audio MLLM, where audio tokens are interleaved within the prompt.<n>Our findings show that while even zero-shot interleaved prompting improves performance on our reasoning tasks, a small amount of fine-tuning improves the results further.
  arXiv  Detail & Related papers  (2025-11-04T03:54:55Z)
• From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data [55.2480439325792]
  Audio-aware large language models (ALLMs) have recently made great strides in understanding and processing audio inputs.<n>These models are typically adapted from text-based large language models (LLMs) through additional training on audio-related tasks.<n>We propose a data generation framework that produces contrastive-like training data, designed to enhance ALLMs' ability to differentiate between present and absent sounds.
  arXiv  Detail & Related papers  (2025-05-26T16:08:41Z)
• Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples [55.2480439325792]
  Recent advancements in audio-aware large language models (ALLMs) enable them to process and understand audio inputs.<n>These models often hallucinate non-existent sound events, reducing their reliability in real-world applications.<n>We propose LISTEN, a contrastive-like training method that enhances ALLMs' ability to distinguish between present and absent sounds.
  arXiv  Detail & Related papers  (2025-05-20T15:44:01Z)
• T-VSL: Text-Guided Visual Sound Source Localization in Mixtures [33.28678401737415]
  We develop a framework to disentangle audio-visual source correspondence from multi-source mixtures. Our framework exhibits promising zero-shot transferability to unseen classes during test time. Experiments conducted on the MUSIC, VGGSound, and VGGSound-Instruments datasets demonstrate significant performance improvements over state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-04-02T09:07:05Z)
• QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition [47.103732403296654]
  Multi-source semantic space can be represented as the Cartesian product of single-source sub-spaces. We introduce a global-to-local quantization mechanism, which distills knowledge from stable global (clip-level) features into local (frame-level) ones. Experiments demonstrate that our semantically decomposed audio representation significantly improves AVS performance.
  arXiv  Detail & Related papers  (2023-09-29T20:48:44Z)
• Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification [55.624946113550195]
  This paper proposes a cross-modal speech co-learning paradigm. Two cross-modal boosters are introduced based on an audio-visual pseudo-siamese structure to learn the modality-transformed correlation. Experimental results on the LRSLip3, GridLip, LomGridLip, and VoxLip datasets demonstrate that our proposed method achieves 60% and 20% average relative performance improvement.
  arXiv  Detail & Related papers  (2023-02-22T10:06:37Z)
• Separate What You Describe: Language-Queried Audio Source Separation [53.65665794338574]
  We introduce the task of language-queried audio source separation (LASS) LASS aims to separate a target source from an audio mixture based on a natural language query of the target source. We propose LASS-Net, an end-to-end neural network that is learned to jointly process acoustic and linguistic information.
  arXiv  Detail & Related papers  (2022-03-28T23:47:57Z)
• Semi-supervised source localization in reverberant environments with deep generative modeling [25.085177610870666]
  A semi-supervised approach to acoustic source localization in reverberant environments is proposed. The approach is based on deep generative modeling. We find that VAE-SSL can outperform both SRP-PHAT and fully supervised CNNs.
  arXiv  Detail & Related papers  (2021-01-26T08:54:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.