Semantic Audio-Visual Navigation in Continuous Environments

• URL: http://arxiv.org/abs/2603.19660v1
• Date: Fri, 20 Mar 2026 05:49:50 GMT
• Title: Semantic Audio-Visual Navigation in Continuous Environments
• Abstract summary: Audio-visual navigation enables embodied agents to navigate toward sound-emitting targets by leveraging both auditory and visual cues.<n>We introduce Audio-Visual Navigation in Continuous Environments (SAVNCE), where agents can move freely in 3D spaces and perceive temporally and spatially coherent audio-visual streams.<n>We propose MAGNet, a multimodal transformer-based model that jointly encodes spatial and semantic goal representations and integrates historical context with self-motion cues to enable memoryaugmented goal reasoning.
• Score: 20.046962371381763
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-visual navigation enables embodied agents to navigate toward sound-emitting targets by leveraging both auditory and visual cues. However, most existing approaches rely on precomputed room impulse responses (RIRs) for binaural audio rendering, restricting agents to discrete grid positions and leading to spatially discontinuous observations. To establish a more realistic setting, we introduce Semantic Audio-Visual Navigation in Continuous Environments (SAVN-CE), where agents can move freely in 3D spaces and perceive temporally and spatially coherent audio-visual streams. In this setting, targets may intermittently become silent or stop emitting sound entirely, causing agents to lose goal information. To tackle this challenge, we propose MAGNet, a multimodal transformer-based model that jointly encodes spatial and semantic goal representations and integrates historical context with self-motion cues to enable memory-augmented goal reasoning. Comprehensive experiments demonstrate that MAGNet significantly outperforms state-of-the-art methods, achieving up to a 12.1\% absolute improvement in success rate. These results also highlight its robustness to short-duration sounds and long-distance navigation scenarios. The code is available at https://github.com/yichenzeng24/SAVN-CE.

Related papers

• Audio Spatially-Guided Fusion for Audio-Visual Navigation [6.536516025407856]
  We propose an Audio Spatially-Guided Fusion for Audio-Visual Navigation method.<n>First, we design an audio spatial feature encoder, which adaptively extracts target-related spatial state information.<n>Based on this, we introduce an Audio Spatial State Guided Fusion (ASGF) to achieve dynamic alignment and adaptive fusion of multimodal features.
  arXiv  Detail & Related papers  (2026-04-02T07:15:17Z)
• NarraScore: Bridging Visual Narrative and Musical Dynamics via Hierarchical Affective Control [59.6128550986024]
  NarraScore is a hierarchical framework predicated on the core insight that emotion serves as a high-density compression of narrative logic.<n>NarraScore employs a Dual-Branch Injection strategy to reconcile global structure with local dynamism.<n>NarraScore achieves state-of-the-art consistency and narrative alignment with negligible computational overhead.
  arXiv  Detail & Related papers  (2026-02-09T09:39:42Z)
• Audio-Guided Visual Perception for Audio-Visual Navigation [38.672625073122276]
  Audio-Visual Embodied Navigation aims to enable agents to autonomously navigate to sound sources in unknown 3D environments using auditory cues.<n>This limitation stems from the lack of explicit alignment mechanisms between auditory signals and corresponding visual regions.<n>We propose the AGVP framework, which transforms sound from policy-memorable acoustic fingerprint cues into spatial guidance.
  arXiv  Detail & Related papers  (2025-10-13T05:06:45Z)
• Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigation [41.85539404067887]
  In audio-visual navigation (AVN) tasks, an embodied agent must autonomously localize a sound source in complex 3D environments.<n>Existing methods often rely on static modality fusion strategies and neglect the spatial cues embedded in stereo audio.<n>We propose an end-to-end reinforcement learning-based AVN framework with two key innovations.
  arXiv  Detail & Related papers  (2025-09-21T05:11:09Z)
• Active Audio-Visual Separation of Dynamic Sound Sources [93.97385339354318]
  We propose a reinforcement learning agent equipped with a novel transformer memory that learns motion policies to control its camera and microphone. We show that our model is able to learn efficient behavior to carry out continuous separation of a time-varying audio target.
  arXiv  Detail & Related papers  (2022-02-02T02:03:28Z)
• Dynamical Audio-Visual Navigation: Catching Unheard Moving Sound Sources in Unmapped 3D Environments [0.0]
  We introduce the novel dynamic audio-visual navigation benchmark in which an embodied AI agent must catch a moving sound source in an unmapped environment in the presence of distractors and noisy sounds. Our approach outperforms the current state-of-the-art with better generalization to unheard sounds and better robustness to noisy scenarios.
  arXiv  Detail & Related papers  (2022-01-12T03:08:03Z)
• Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Sounds [5.002862602915434]
  Audio-visual navigation combines sight and hearing to navigate to a sound-emitting source in an unmapped environment. We propose the novel dynamic audio-visual navigation benchmark which requires to catch a moving sound source in an environment with noisy and distracting sounds. We demonstrate that our approach consistently outperforms the current state-of-the-art by a large margin across all tasks of moving sounds, unheard sounds, and noisy environments.
  arXiv  Detail & Related papers  (2021-11-29T15:17:46Z)
• Move2Hear: Active Audio-Visual Source Separation [90.16327303008224]
  We introduce the active audio-visual source separation problem, where an agent must move intelligently in order to better isolate the sounds coming from an object of interest. We introduce a reinforcement learning approach that trains movement policies controlling the agent's camera and microphone placement over time. We demonstrate our model's ability to find minimal movement sequences with maximal payoff for audio source separation.
  arXiv  Detail & Related papers  (2021-05-15T04:58:08Z)
• Semantic Audio-Visual Navigation [93.12180578267186]
  We introduce semantic audio-visual navigation, where objects in the environment make sounds consistent with their semantic meaning. We propose a transformer-based model to tackle this new semantic AudioGoal task. Our method strongly outperforms existing audio-visual navigation methods by learning to associate semantic, acoustic, and visual cues.
  arXiv  Detail & Related papers  (2020-12-21T18:59:04Z)
• Learning to Set Waypoints for Audio-Visual Navigation [89.42192208471735]
  In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source. Existing models learn to act at a fixed granularity of agent motion and rely on simple recurrent aggregations of the audio observations. We introduce a reinforcement learning approach to audio-visual navigation with two key novel elements.
  arXiv  Detail & Related papers  (2020-08-21T18:00:33Z)
• Temporal-Spatial Neural Filter: Direction Informed End-to-End Multi-channel Target Speech Separation [66.46123655365113]
  Target speech separation refers to extracting the target speaker's speech from mixed signals. Two main challenges are the complex acoustic environment and the real-time processing requirement. We propose a temporal-spatial neural filter, which directly estimates the target speech waveform from multi-speaker mixture.
  arXiv  Detail & Related papers  (2020-01-02T11:12:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.