Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation

• URL: http://arxiv.org/abs/2608.29121v1
• Date: Sat, 29 Aug 2026 08:02:27 GMT
• Title: Acoustically Grounded Cost Learning for Open-Vocabulary Audio-Visual Semantic Segmentation
• Abstract summary: Open-Vocabulary Audio-Visual Semantic (OV-AVSS) aims to perform pixel-level segmentation of sound-emitting objects from an open set of categories.<n>We reformulate the objective to be category-specific and propose a novel Acoustically Grounded Cost Learning framework.<n>Experiments on the AVSBench-OV dataset demonstrate that our method significantly outperforms previous state-of-the-art approaches.
• Score: 95.88527433039519
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-Vocabulary Audio-Visual Semantic Segmentation (OV-AVSS) aims to perform pixel-level segmentation of sound-emitting objects from an open set of categories. The previous method relies on a class-agnostic foreground definition, which groups semantically diverse objects into a heterogeneous positive set, causing the model to learn unstable sounding patterns and produce unreliable proposals. To address this, we reformulate the objective to be category-specific and propose a novel Acoustically Grounded Cost Learning (AGCL) framework to transform the static, audio-agnostic visual-text priors into dynamic, audio-grounded cost representations. For intra-category soundingness discovery, we devise Audio-Modulated Cost Generation (AMCG) and Audio-Guided Temporal Aggregation (AGTA) modules to enable both frame-level sounding region highlighting and video-level temporal refinement with a low-intrusive audio injection mechanism. For inter-category distractor discrimination, we introduce a Synergistic Distractor Mining (SDM) strategy, which selectively penalizes acoustically and semantically confusing negative categories to learn more discriminative decision boundaries. Extensive experiments on the AVSBench-OV dataset demonstrate that our method significantly outperforms previous state-of-the-art approaches, particularly on unseen categories. Code is available at https://github.com/spyflying/AGCL.

Related papers

• Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization [69.67186845167568]
  Open-vocabulary audio-visual event localization (OV-AVEL) jointly models audio-visual cues to recognize and temporally localize events.<n>Existing methods primarily learn joint audio-visual representations in Euclidean space.<n>We propose a hierarchical semantic constrained heterogeneous graph (HSCHG) for audio-visual event localization framework.
  arXiv  Detail & Related papers  (2026-06-05T08:23:58Z)
• Shared Representation Learning for Reference-Guided Targeted Sound Detection [12.339689044414401]
  We propose a unified encoder architecture that processes both the reference and mixture audio within a shared representation space.<n>We establish a new state-of-the-art benchmark for targeted sound detection with a segment-level F1 score of 83.15%.
  arXiv  Detail & Related papers  (2026-03-17T18:05:21Z)
• Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation [59.11043512784162]
  We propose SDAVS, equipped with the Selective Noise-Resilient Processor (SNRP) module and the Discriminative Audio-Visual Mutual Fusion (DAMF) strategy.<n> Experimental results demonstrate that our proposed method achieves state-of-the-art performance on benchmark AVS datasets.
  arXiv  Detail & Related papers  (2026-03-15T03:22:24Z)
• Implicit Counterfactual Learning for Audio-Visual Segmentation [50.69377287012591]
  We propose the implicit counterfactual framework (ICF) to achieve unbiased cross-modal understanding.<n>Due to the lack of semantics, heterogeneous representations may lead to erroneous matches.<n>We introduce the multi-granularity implicit text (MIT) involving video-, segment- and frame-level as the bridge to establish the modality-shared space.
  arXiv  Detail & Related papers  (2025-07-28T11:46:35Z)
• $C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction [80.57232374640911]
  We propose a model-agnostic strategy called the Mask-And-Recover (MAR)<n>MAR integrates both inter- and intra-modality contextual correlations to enable global inference within extraction modules.<n>To better target challenging parts within each sample, we introduce a Fine-grained Confidence Score (FCS) model.
  arXiv  Detail & Related papers  (2025-04-01T13:01:30Z)
• Do Audio-Visual Segmentation Models Truly Segment Sounding Objects? [38.98706069359109]
  We introduce AVSBench-Robust, a benchmark incorporating diverse negative audio scenarios including silence, ambient noise, and off-screen sounds.<n>Our approach achieves remarkable improvements in both standard metrics and robustness measures, maintaining near-perfect false positive rates.
  arXiv  Detail & Related papers  (2025-02-01T07:40:29Z)
• Open-Vocabulary Audio-Visual Semantic Segmentation [12.330521982730836]
  We introduce a new task: open-vocabulary audio-visual semantic segmentation, extending AVSS task to open-world scenarios beyond the annotated label space. We propose the first open-vocabulary AVSS framework, OV-AVSS, which mainly consists of two parts: 1) a universal sound source localization module to perform audio-visual fusion and locate all potential sounding objects and 2) an open-vocabulary classification module to predict categories with the help of the prior knowledge from large-scale pre-trained vision-language models.
  arXiv  Detail & Related papers  (2024-07-31T16:14:09Z)
• DiffSED: Sound Event Detection with Denoising Diffusion [70.18051526555512]
  We reformulate the SED problem by taking a generative learning perspective. Specifically, we aim to generate sound temporal boundaries from noisy proposals in a denoising diffusion process. During training, our model learns to reverse the noising process by converting noisy latent queries to the groundtruth versions.
  arXiv  Detail & Related papers  (2023-08-14T17:29:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.