From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models

• URL: http://arxiv.org/abs/2606.25391v1
• Date: Wed, 24 Jun 2026 04:42:57 GMT
• Title: From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models
• Abstract summary: Real-world auditory interpretation requires Context-Aware Auditory Scene Understanding (CASU)<n>We introduce the CASU benchmark, which assesses whether Audio LLMs can interpret auditory scenes composed of speech, acoustic events, and background environments.<n>We propose a scalable pipeline for constructing time-accurate, semi-synthetic audio streams by composing real-world scene sounds with synthetic speech.
• Score: 19.687157920670987
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent Large Audio Language Models (LALMs) have achieved remarkable progress in audio perceptual tasks across individual acoustic layers, including speech, sound, and music. However, existing benchmarks predominantly evaluate these layers in isolation, overlooking the complex contextual relationships that arise when multiple acoustic sources co-occur in real-world auditory scenes. Real-world auditory interpretation requires Context-Aware Auditory Scene Understanding (CASU): the ability to comprehend the holistic scene by integrating sound layers. To evaluate this capability, we introduce the CASU benchmark, which assesses whether Audio LLMs can interpret auditory scenes composed of speech, acoustic events (e.g., announcements), and background environments (e.g., traffic), and reason about the logical relationships between these layers. We propose a scalable pipeline for constructing time-accurate, semi-synthetic audio streams by composing real-world scene sounds with synthetic speech. Building on this data, we design four tasks that probe scene understanding: contextual question answering, entity extraction from the scene, speaker role inference, and counterfactual reasoning where scene is manipulated. Experiments across multiple LALMs demonstrate that effective auditory scene understanding requires integration over all auditory layers, rather than reliance on speech or sound alone, underscoring the necessity of CASU for advancing complex audio understanding in LALMs.

Related papers

• Do Audio-Visual Large Language Models Really See and Hear? [66.8531220331223]
  We analyze how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs.<n>Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.
  arXiv  Detail & Related papers  (2026-04-03T00:48:49Z)
• How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation [97.0235251827591]
  Large language models (LLMs) have been widely used as knowledge backbones of Large Audio Language Models (LALMs)<n>We study how much auditory knowledge they encode through text-only pre-training and how this affects downstream performance.<n>Our findings reveal that auditory knowledge varies substantially across families, and text-only results are strongly correlated with audio performance.
  arXiv  Detail & Related papers  (2026-03-19T17:50:07Z)
• Causal Tracing of Audio-Text Fusion in Large Audio Language Models [48.849764961130795]
  We adapt causal tracing to investigate the internal information flow of LALMs during audio comprehension.<n>By conducting layer-wise and token-wise analyses across DeSTA, Qwen, and Voxtral, we evaluate the causal effects of individual hidden states.
  arXiv  Detail & Related papers  (2026-03-14T05:40:59Z)
• Echo: Towards Advanced Audio Comprehension via Audio-Interleaved Reasoning [39.264735719707154]
  Current efforts replicate text-based reasoning by contextualizing audio content through a one-time encoding.<n>We propose audio-interleaved reasoning to break through this bottleneck.<n>We present Echo, a LALM capable of dynamically re-listening to audio in demand during reasoning.
  arXiv  Detail & Related papers  (2026-02-12T13:06:34Z)
• WoW-Bench: Evaluating Fine-Grained Acoustic Perception in Audio-Language Models via Marine Mammal Vocalizations [67.6147632074449]
  We introduce the World-of-Whale benchmark (WoW-Bench) to evaluate low-level auditory perception and cognition using marine mammal vocalizations.<n>WoW-Bench is composed of a Perception benchmark for categorizing novel sounds and a Cognition benchmark, inspired by Bloom's taxonomy, to assess the abilities to remember, understand, apply, and analyze sound events.<n>Experiments with state-of-the-art LALMs show performance far below human levels, indicating a need for stronger auditory grounding in LALMs.
  arXiv  Detail & Related papers  (2025-08-28T16:29:46Z)
• Auditory Intelligence: Understanding the World Through Sound [4.6684925321613076]
  I propose a conceptual reframing of auditory intelligence as a layered, situated process that encompasses perception, reasoning, and interaction.<n>I introduce four cognitively inspired task paradigms-ASPIRE, SODA, AUX, and AUGMENT-those structure auditory understanding across time-frequency pattern captioning, hierarchical event/scene description, causal explanation, and goal-driven interpretation.
  arXiv  Detail & Related papers  (2025-08-11T10:25:58Z)
• Evaluation of Deep Audio Representations for Hearables [1.5646349560044959]
  This dataset includes 1,158 audio tracks, each 30 seconds long, created by spatially mixing proprietary monologues with high-quality recordings of everyday acoustic scenes.<n>Our benchmark encompasses eight tasks that assess the general context, speech sources, and technical acoustic properties of the audio scenes.<n>This superiority underscores the advantage of models trained on diverse audio collections, confirming their applicability to a wide array of auditory tasks, including encoding the environment properties necessary for hearable steering.
  arXiv  Detail & Related papers  (2025-02-10T16:51:11Z)
• Self-Supervised Visual Acoustic Matching [63.492168778869726]
  Acoustic matching aims to re-synthesize an audio clip to sound as if it were recorded in a target acoustic environment. We propose a self-supervised approach to visual acoustic matching where training samples include only the target scene image and audio. Our approach jointly learns to disentangle room acoustics and re-synthesize audio into the target environment, via a conditional GAN framework and a novel metric.
  arXiv  Detail & Related papers  (2023-07-27T17:59:59Z)
• AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis [61.07542274267568]
  We study a new task -- real-world audio-visual scene synthesis -- and a first-of-its-kind NeRF-based approach for multimodal learning. We propose an acoustic-aware audio generation module that integrates prior knowledge of audio propagation into NeRF. We present a coordinate transformation module that expresses a view direction relative to the sound source, enabling the model to learn sound source-centric acoustic fields.
  arXiv  Detail & Related papers  (2023-02-04T04:17:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.