Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

• URL: http://arxiv.org/abs/2608.09435v1
• Date: Mon, 10 Aug 2026 11:06:28 GMT
• Title: Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models
• Abstract summary: We present a benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources.<n>It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning.<n>Building on this benchmark we propose ST- Omni-R1, which achieves FOA- semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning.
• Score: 41.044279131530885
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding dynamic sound sources requires jointly determining what produces a sound, where the source is located, and how it moves over time. Yet existing audio-language models often represent clips as global acoustic events, while vision-language models lack the spatial audio cues needed to localize and track individual sources. To evaluate this missing capability, we introduce ST-OmniQA, a spatio-temporal audio-visual question-answering benchmark built from panoramic videos paired with synchronized first-order Ambisonics (FOA) audio of moving sound sources. It contains 40K videos and 400K question-answer pairs organized into four capability levels covering sound-event recognition, direction of arrival, source distance, motion trajectories, and temporally grounded audio-visual reasoning. Building on this benchmark, we propose ST-Omni-R1, which integrates FOA-derived semantic and trajectory representations with panoramic visual context and is trained through progressive curriculum learning and reasoning-tree reinforcement learning. ST-Omni-R1 achieves 77.83\% average semantic accuracy across the four levels, compared with 37.28\% for the best evaluated baseline. Results on three public spatial-audio benchmarks further indicate that its learned spatial and motion representations transfer beyond ST-OmniQA.

Related papers

• Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources [57.68713138159972]
  We present a time-resolved audio encoder that learns event semantics together with source trajectories.<n>Experiments show that this representation improves the semanticlocalization and yields stronger reasoning than spatial and localization-oriented tradeoffs.
  arXiv  Detail & Related papers  (2026-06-12T05:58:31Z)
• SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Cases [27.340743922132067]
  SCENEBench targets a broad form of audio comprehension across four real-world categories: background sound understanding, noise localization, cross-linguistic speech understanding, and vocal characterizer recognition.<n>The purpose of this benchmark suite is to assess audio beyond just what words are said - rather, how they are said and the non-speech components of the audio.<n>We assess five state-of-the-art LALMs and find critical gaps: performance varies across tasks, with some tasks performing below random chance and others achieving high accuracy.
  arXiv  Detail & Related papers  (2026-03-10T16:15:12Z)
• STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence [81.94084852268468]
  We formalize audio 4D intelligence that is defined as reasoning over sound dynamics in time and 3D space.<n> STAR-Bench combines a Foundational Acoustic Perception setting with a Holistic Spatio-Temporal Reasoning setting.<n>Our data curation pipeline uses two methods to ensure high-quality samples.
  arXiv  Detail & Related papers  (2025-10-28T17:50:34Z)
• Sci-Phi: A Large Language Model Spatial Audio Descriptor [25.302416479626974]
  Sci-Phi is a spatial audio model with dual spatial and spectral encoders.<n>It enumerates and describes up to four directional sound sources in one pass.<n>It generalizes to real room impulse responses with only minor performance degradation.
  arXiv  Detail & Related papers  (2025-10-07T03:06:02Z)
• In-the-wild Audio Spatialization with Flexible Text-guided Localization [37.60344400859993]
  To enhance immersive experiences, audio offers spatial awareness of sounding objects in AR, VR, and embodied AI applications.<n>While existing audio spatialization methods can generally map any available monaural audio to audio signals, they often lack the flexible and interactive control needed in complex multi-object user-interactive environments.<n>We propose a Text-guided Audio Spatialization (TAS) framework that utilizes flexible text prompts and evaluates our model from unified generation and comprehension perspectives.
  arXiv  Detail & Related papers  (2025-06-01T09:41:56Z)
• Solution for Temporal Sound Localisation Task of ECCV Second Perception Test Challenge 2024 [3.4947857354806633]
  This report proposes an improved method for the Temporal Sound Localisation task. It localizes and classifies the sound events occurring in the video according to a predefined set of sound classes. Our approach ranks first in the final test with a score of 0.4925.
  arXiv  Detail & Related papers  (2024-09-29T07:28:21Z)
• AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis [62.33446681243413]
  view acoustic synthesis aims to render audio at any target viewpoint, given a mono audio emitted by a sound source at a 3D scene.<n>Existing methods have proposed NeRF-based implicit models to exploit visual cues as a condition for synthesizing audio.<n>We propose a novel Audio-Visual Gaussian Splatting (AV-GS) model to characterize the entire scene environment.<n>Experiments validate the superiority of our AV-GS over existing alternatives on the real-world RWAS and simulation-based SoundSpaces datasets.
  arXiv  Detail & Related papers  (2024-06-13T08:34:12Z)
• Self-Supervised Visual Acoustic Matching [63.492168778869726]
  Acoustic matching aims to re-synthesize an audio clip to sound as if it were recorded in a target acoustic environment. We propose a self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio. Our approach jointly learns to disentangle room acoustics and re-synthesize audio into the target environment, via a conditional GAN framework and a novel metric.
  arXiv  Detail & Related papers  (2023-07-27T17:59:59Z)
• AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis [61.07542274267568]
  We study a new task -- real-world audio-visual scene synthesis -- and a first-of-its-kind NeRF-based approach for multimodal learning. We propose an acoustic-aware audio generation module that integrates prior knowledge of audio propagation into NeRF. We present a coordinate transformation module that expresses a view direction relative to the sound source, enabling the model to learn sound source-centric acoustic fields.
  arXiv  Detail & Related papers  (2023-02-04T04:17:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.