Spatial Audio Motion Understanding and Reasoning

• URL: http://arxiv.org/abs/2509.14666v1
• Date: Thu, 18 Sep 2025 06:53:22 GMT
• Title: Spatial Audio Motion Understanding and Reasoning
• Authors: Arvind Krishna Sridhar, Yinyi Guo, Erik Visser,
• Abstract summary: spatial audio reasoning enables machines to interpret auditory scenes by understanding events and their spatial attributes.<n>We introduce a spatial audio encoder that processes spatial audio to detect multiple overlapping events and estimate their spatial attributes, Direction of Arrival (DoA) and source distance, at the frame level.<n>Second, to answer complex queries about dynamic audio scenes involving moving sources, we condition a large language model (LLM) on structured spatial attributes extracted by our model.
• Score: 8.029049649310211
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spatial audio reasoning enables machines to interpret auditory scenes by understanding events and their spatial attributes. In this work, we focus on spatial audio understanding with an emphasis on reasoning about moving sources. First, we introduce a spatial audio encoder that processes spatial audio to detect multiple overlapping events and estimate their spatial attributes, Direction of Arrival (DoA) and source distance, at the frame level. To generalize to unseen events, we incorporate an audio grounding model that aligns audio features with semantic audio class text embeddings via a cross-attention mechanism. Second, to answer complex queries about dynamic audio scenes involving moving sources, we condition a large language model (LLM) on structured spatial attributes extracted by our model. Finally, we introduce a spatial audio motion understanding and reasoning benchmark dataset and demonstrate our framework's performance against the baseline model.

Related papers

• SpatialV2A: Visual-Guided High-fidelity Spatial Audio Generation [15.901895888187711]
  BinauralVGGSound is the first large-scale video-binaural audio dataset designed to support spatially aware video-to-audio generation.<n>Our framework incorporates a visual-guided audio spatialization module that ensures the generated audio exhibits realistic spatial attributes and layered spatial depth.
  arXiv  Detail & Related papers  (2026-01-21T14:14:37Z)
• AudioScene: Integrating Object-Event Audio into 3D Scenes [19.66595321540055]
  We present two novel audiospatial scene datasets, AudioScanNet and AudioRoboTHOR.<n>By integrating audio clips with spatially aligned 3D scenes, our datasets enable research on how audio signals interact with spatial context.
  arXiv  Detail & Related papers  (2025-11-25T14:28:13Z)
• Sci-Phi: A Large Language Model Spatial Audio Descriptor [25.302416479626974]
  Sci-Phi is a spatial audio model with dual spatial and spectral encoders.<n>It enumerates and describes up to four directional sound sources in one pass.<n>It generalizes to real room impulse responses with only minor performance degradation.
  arXiv  Detail & Related papers  (2025-10-07T03:06:02Z)
• Audio Does Matter: Importance-Aware Multi-Granularity Fusion for Video Moment Retrieval [58.640807985155554]
  Video Moment Retrieval (VMR) aims to retrieve a specific moment semantically related to a given query.<n>Most existing VMR methods solely focus on the visual and textual modalities while neglecting the complementary but important audio modality.<n>We propose a novel Importance-aware Multi-Granularity fusion model (IMG), which learns to dynamically and selectively aggregate the audio-vision-text contexts for VMR.
  arXiv  Detail & Related papers  (2025-08-06T09:58:43Z)
• SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generation [50.03810359300705]
  SpA2V decomposes the generation process into two stages: audio-guided video planning and layout-grounded video generation.<n>We show that SpA2V excels in generating realistic videos with semantic and spatial alignment to the input audios.
  arXiv  Detail & Related papers  (2025-08-01T17:05:04Z)
• ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing [52.33281620699459]
  ThinkSound is a novel framework that leverages Chain-of-Thought (CoT) reasoning to enable stepwise, interactive audio generation and editing for videos.<n>Our approach decomposes the process into three complementary stages: semantically coherent, interactive object-centric refinement through precise user interactions, and targeted editing guided by natural language instructions.<n> Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics.
  arXiv  Detail & Related papers  (2025-06-26T16:32:06Z)
• DualSpec: Text-to-spatial-audio Generation via Dual-Spectrogram Guided Diffusion Model [46.60765174200236]
  We propose a text-to-spatial-audio (TTSA) generation framework named DualSpec.<n>It first trains variational autoencoders (VAEs) for extracting the latent acoustic representations from sound event audio.<n>Finally, it trains a diffusion model from the latent acoustic representations and text features for the spatial audio generation.
  arXiv  Detail & Related papers  (2025-02-26T09:01:59Z)
• Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation [32.24603883810094]
  Controlling stereo audio with spatial contexts remains challenging due to high data costs and unstable generative models.<n>We first construct a large-scale, simulation-based, and GPT-assisted dataset, BEWO-1M, with abundant soundscapes and descriptions.<n>By leveraging spatial guidance, our model achieves the objective of generating immersive and controllable spatial audio from text.
  arXiv  Detail & Related papers  (2024-10-14T16:18:29Z)
• BAT: Learning to Reason about Spatial Sounds with Large Language Models [45.757161909533714]
  We present BAT, which combines the sound perception ability of a spatial scene analysis model with the natural language reasoning capabilities of a large language model (LLM)<n>Our experiments demonstrate BAT's superior performance on both spatial sound perception and reasoning.
  arXiv  Detail & Related papers  (2024-02-02T17:34:53Z)
• Audio-Visual Spatial Integration and Recursive Attention for Robust Sound Source Localization [13.278494654137138]
  Humans utilize both audio and visual modalities as spatial cues to locate sound sources. We propose an audio-visual spatial integration network that integrates spatial cues from both modalities. Our method can perform more robust sound source localization.
  arXiv  Detail & Related papers  (2023-08-11T11:57:58Z)
• Separate Anything You Describe [53.30484933564858]
  Language-queried audio source separation (LASS) is a new paradigm for computational auditory scene analysis (CASA)<n>AudioSep is a foundation model for open-domain audio source separation with natural language queries.
  arXiv  Detail & Related papers  (2023-08-09T16:09:44Z)
• Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Video [94.42811508809994]
  We propose an audio spatialization method that draws on visual information in videos to convert their monaural (single-channel) audio to audio. Whereas existing approaches leverage visual features extracted directly from video frames, our approach explicitly disentangles the geometric cues present in the visual stream to guide the learning process.
  arXiv  Detail & Related papers  (2021-11-21T19:26:45Z)
• Data Fusion for Audiovisual Speaker Localization: Extending Dynamic Stream Weights to the Spatial Domain [103.3388198420822]
  Esting the positions of multiple speakers can be helpful for tasks like automatic speech recognition or speaker diarization. This paper proposes a novel audiovisual data fusion framework for speaker localization by assigning individual dynamic stream weights to specific regions. A performance evaluation using audiovisual recordings yields promising results, with the proposed fusion approach outperforming all baseline models.
  arXiv  Detail & Related papers  (2021-02-23T09:59:31Z)
• Learning Representations from Audio-Visual Spatial Alignment [76.29670751012198]
  We introduce a novel self-supervised pretext task for learning representations from audio-visual content. The advantages of the proposed pretext task are demonstrated on a variety of audio and visual downstream tasks.
  arXiv  Detail & Related papers  (2020-11-03T16:20:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.