Audio Spatially-Guided Fusion for Audio-Visual Navigation

• URL: http://arxiv.org/abs/2604.02389v1
• Date: Thu, 02 Apr 2026 07:15:17 GMT
• Title: Audio Spatially-Guided Fusion for Audio-Visual Navigation
• Abstract summary: We propose an Audio Spatially-Guided Fusion for Audio-Visual Navigation method.<n>First, we design an audio spatial feature encoder, which adaptively extracts target-related spatial state information.<n>Based on this, we introduce an Audio Spatial State Guided Fusion (ASGF) to achieve dynamic alignment and adaptive fusion of multimodal features.
• Score: 6.536516025407856
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Audio-visual Navigation refers to an agent utilizing visual and auditory information in complex 3D environments to accomplish target localization and path planning, thereby achieving autonomous navigation. The core challenge of this task lies in the following: how the agent can break free from the dependence on training data and achieve autonomous navigation with good generalization performance when facing changes in environments and sound sources. To address this challenge, we propose an Audio Spatially-Guided Fusion for Audio-Visual Navigation method. First, we design an audio spatial feature encoder, which adaptively extracts target-related spatial state information through an audio intensity attention mechanism; based on this, we introduce an Audio Spatial State Guided Fusion (ASGF) to achieve dynamic alignment and adaptive fusion of multimodal features, effectively alleviating noise interference caused by perceptual uncertainty. Experimental results on the Replica and Matterport3D datasets indicate that our method is particularly effective on unheard tasks, demonstrating improved generalization under unknown sound source distributions.

Related papers

• The Attention Triangle in Audio-Video Models [67.46972888113658]
  Audio-video diffusion models rely on cross-modal attention to coordinate text, sound, and visual content, yet this same mechanism can introduce subtle and systematic semantic leakage.<n>We study these models by probing and analyzing the attention triangle'' comprising the three cross-attention edges connecting the text, audio, and video streams.<n>Our analysis reveals that routing along the audio-video edge is bidirectional: audio can influence video generation, while video can influence audio generation.
  arXiv  Detail & Related papers  (2026-09-03T09:33:27Z)
• Semantic Audio-Visual Navigation in Continuous Environments [20.046962371381763]
  Audio-visual navigation enables embodied agents to navigate toward sound-emitting targets by leveraging both auditory and visual cues.<n>We introduce Audio-Visual Navigation in Continuous Environments (SAVNCE), where agents can move freely in 3D spaces and perceive temporally and spatially coherent audio-visual streams.<n>We propose MAGNet, a multimodal transformer-based model that jointly encodes spatial and semantic goal representations and integrates historical context with self-motion cues to enable memoryaugmented goal reasoning.
  arXiv  Detail & Related papers  (2026-03-20T05:49:50Z)
• Selective Noise Suppression and Discriminative Mutual Interaction for Robust Audio-Visual Segmentation [59.11043512784162]
  We propose SDAVS, equipped with the Selective Noise-Resilient Processor (SNRP) module and the Discriminative Audio-Visual Mutual Fusion (DAMF) strategy.<n> Experimental results demonstrate that our proposed method achieves state-of-the-art performance on benchmark AVS datasets.
  arXiv  Detail & Related papers  (2026-03-15T03:22:24Z)
• Audio-Guided Visual Perception for Audio-Visual Navigation [38.672625073122276]
  Audio-Visual Embodied Navigation aims to enable agents to autonomously navigate to sound sources in unknown 3D environments using auditory cues.<n>This limitation stems from the lack of explicit alignment mechanisms between auditory signals and corresponding visual regions.<n>We propose the AGVP framework, which transforms sound from policy-memorable acoustic fingerprint cues into spatial guidance.
  arXiv  Detail & Related papers  (2025-10-13T05:06:45Z)
• Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigation [41.85539404067887]
  In audio-visual navigation (AVN) tasks, an embodied agent must autonomously localize a sound source in complex 3D environments.<n>Existing methods often rely on static modality fusion strategies and neglect the spatial cues embedded in stereo audio.<n>We propose an end-to-end reinforcement learning-based AVN framework with two key innovations.
  arXiv  Detail & Related papers  (2025-09-21T05:11:09Z)
• Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization [50.122441710500055]
  We present LoCo, a Locality-aware cross-modal Correspondence learning framework for Audio-Visual Events (DAVE)<n>LoCo applies Local Correspondence Feature (LCF) Modulation to enforce unimodal encoders to focus on modality-shared semantics.<n>We further customize Local Adaptive Cross-modal (LAC) Interaction, which dynamically adjusts attention regions in a data-driven manner.
  arXiv  Detail & Related papers  (2024-09-12T11:54:25Z)
• CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments [41.21509045214965]
  CAVEN is a framework in which the agent may interact with a human/oracle for solving the task of navigating to an audio goal. Our results show that our fully-conversational approach leads to nearly an order-of-magnitude improvement in success rate.
  arXiv  Detail & Related papers  (2023-06-06T22:32:49Z)
• AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments [60.98664330268192]
  We present AVLEN -- an interactive agent for Audio-Visual-Language Embodied Navigation. The goal of AVLEN is to localize an audio event via navigating the 3D visual world. To realize these abilities, AVLEN uses a multimodal hierarchical reinforcement learning backbone.
  arXiv  Detail & Related papers  (2022-10-14T16:35:06Z)
• Pay Self-Attention to Audio-Visual Navigation [24.18976027602831]
  We propose an end-to-end framework to learn chasing after a moving audio target using a context-aware audio-visual fusion strategy. Our thorough experiments validate the superior performance of FSAAVN in comparison with the state-of-the-arts.
  arXiv  Detail & Related papers  (2022-10-04T03:42:36Z)
• Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain [103.3388198420822]
  Esting the positions of multiple speakers can be helpful for tasks like automatic speech recognition or speaker diarization. This paper proposes a novel audiovisual data fusion framework for speaker localization by assigning individual dynamic stream weights to specific regions. A performance evaluation using audiovisual recordings yields promising results, with the proposed fusion approach outperforming all baseline models.
  arXiv  Detail & Related papers  (2021-02-23T09:59:31Z)
• Semantic Audio-Visual Navigation [93.12180578267186]
  We introduce semantic audio-visual navigation, where objects in the environment make sounds consistent with their semantic meaning. We propose a transformer-based model to tackle this new semantic AudioGoal task. Our method strongly outperforms existing audio-visual navigation methods by learning to associate semantic, acoustic, and visual cues.
  arXiv  Detail & Related papers  (2020-12-21T18:59:04Z)
• Learning to Set Waypoints for Audio-Visual Navigation [89.42192208471735]
  In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source. Existing models learn to act at a fixed granularity of agent motion and rely on simple recurrent aggregations of the audio observations. We introduce a reinforcement learning approach to audio-visual navigation with two key novel elements.
  arXiv  Detail & Related papers  (2020-08-21T18:00:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.