What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability

• URL: http://arxiv.org/abs/2609.12663v1
• Date: Fri, 11 Sep 2026 10:08:29 GMT
• Title: What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability
• Abstract summary: We introduce STAG, to our knowledge the first post-hoc framework for token-level spectro-temporal grounding of captions generated by audio-based MLLMs.<n>STAG estimates the temporal support for each generated token using target-token-specific vocabulary projections of the encoded audio representations.
• Score: 6.338357793019971
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio-based Multimodal Large Language Models (MLLMs) can generate detailed natural-language descriptions of complex acoustic scenes, yet it remains unclear which parts of the input audio support each generated token. This is particularly challenging because acoustic evidence is distributed across time and frequency, and concurrent sound events may overlap temporally while occupying different spectral regions. We introduce STAG, to our knowledge the first post-hoc framework for token-level spectro-temporal grounding of captions generated by audio-based MLLMs. STAG estimates the temporal support for each generated token using target-token-specific vocabulary projections of the encoded audio representations, measures frequency-band relevance through controlled spectral occlusion, and combines the two signals into a spectro-temporal relevance map. We evaluate STAG against ten post-hoc explanation methods across four grounding benchmarks, where it achieves the best event-localization performance on every dataset, and apply it to eight audio-language backbones without parameter updates. Counterfactual deletion further shows that removing the identified evidence selectively reduces confidence in the corresponding event and frequently removes it from the regenerated caption. These results provide behavioral support for the faithfulness and selectivity of the explanations.

Related papers

• Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception [26.919697203335343]
  Large Audio-Language Models (LALMs) have substantially advanced general audio understanding, yet they remain limited in fine-grained temporal perception.<n>We augment the LALM with a dedicated frame-level grounding model while leveraging its semantic modeling capability to represent the event query.<n>Experiments across diverse temporal grounding benchmarks demonstrate strong and consistent improvements over existing methods.
  arXiv  Detail & Related papers  (2026-09-14T08:35:35Z)
• Hear to See: Discerning Stateful Listening for Audio-Visual Instance Segmentation [58.79612575170589]
  Existing methods struggle to match overlapping acoustic events with visual instances and handle audio-visual dynamics.<n>This paper proposes Hear to See (H2S), addressing these challenges through two mechanisms.<n>The code will be open-sourced once the paper is accepted.
  arXiv  Detail & Related papers  (2026-08-04T07:35:57Z)
• Spatio-Temporal Audio Language Modeling for Dynamic Sound Sources [57.68713138159972]
  We present a time-resolved audio encoder that learns event semantics together with source trajectories.<n>Experiments show that this representation improves the semanticlocalization and yields stronger reasoning than spatial and localization-oriented tradeoffs.
  arXiv  Detail & Related papers  (2026-06-12T05:58:31Z)
• Hierarchical Semantic-Constrained Heterogeneous Graph for Audio-Visual Event Localization [69.67186845167568]
  Open-vocabulary audio-visual event localization (OV-AVEL) jointly models audio-visual cues to recognize and temporally localize events.<n>Existing methods primarily learn joint audio-visual representations in Euclidean space.<n>We propose a hierarchical semantic constrained heterogeneous graph (HSCHG) for audio-visual event localization framework.
  arXiv  Detail & Related papers  (2026-06-05T08:23:58Z)
• SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Models [11.23346080242434]
  Large audio language models (LALMs) extend large language models with an audio encoder and large-scale audio data.<n>We propose Spectrotemporal Counting (SpectCount), a data-efficient fine-tuning approach based on fully synthetic audio signals generated on-the-fly.
  arXiv  Detail & Related papers  (2026-06-05T04:50:34Z)
• Temporal Contrastive Decoding: A Training-Free Method for Large Audio-Language Models [56.91801348360746]
  Large audio-language models (LALMs) generalize across speech, sound, and music.<n> unified decoders can exhibit a emphtemporal smoothing bias<n>We propose emphTemporal Contrastive Decoding (TCD), a training-free decoding method for unified LALMs.
  arXiv  Detail & Related papers  (2026-04-16T02:30:41Z)
• Causal Tracing of Audio-Text Fusion in Large Audio Language Models [48.849764961130795]
  We adapt causal tracing to investigate the internal information flow of LALMs during audio comprehension.<n>By conducting layer-wise and token-wise analyses across DeSTA, Qwen, and Voxtral, we evaluate the causal effects of individual hidden states.
  arXiv  Detail & Related papers  (2026-03-14T05:40:59Z)
• Detect Any Sound: Open-Vocabulary Sound Event Detection with Multi-Modal Queries [18.147981850263708]
  We propose a query-based framework for open-vocabulary SED guided by multi-modal queries.<n>DASM formulates SED as a frame-level retrieval task, where audio features are matched against query vectors from text or audio prompts.<n>DASM effectively balances localization accuracy with generalization to novel classes, outperforming CLAP-based methods in open-vocabulary setting.
  arXiv  Detail & Related papers  (2025-07-22T08:24:01Z)
• Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding [36.20990265600332]
  We introduce PU-VALOR, a comprehensive audio-visual dataset comprising over 114,000 pseudo-untrimmed videos with detailed temporal annotations.<n> PU-VALOR is derived from the large-scale but coarse-annotated audio-visual dataset VALOR, through a subtle method involving event-based video clustering.<n>We develop AVicuna, a model capable of aligning audio-visual events with temporal intervals and corresponding text tokens.
  arXiv  Detail & Related papers  (2024-03-24T19:50:49Z)
• QDFormer: Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition [47.103732403296654]
  Multi-source semantic space can be represented as the Cartesian product of single-source sub-spaces. We introduce a global-to-local quantization mechanism, which distills knowledge from stable global (clip-level) features into local (frame-level) ones. Experiments demonstrate that our semantically decomposed audio representation significantly improves AVS performance.
  arXiv  Detail & Related papers  (2023-09-29T20:48:44Z)
• Leveraging Language Model Capabilities for Sound Event Detection [10.792576135806623]
  We propose an end-to-end framework for understanding audio features while simultaneously generating sound event and their temporal location. Specifically, we employ pretrained acoustic models to capture discriminative features across different categories and language models for autoregressive text generation.
  arXiv  Detail & Related papers  (2023-08-22T15:59:06Z)
• Audio-visual multi-channel speech separation, dereverberation and recognition [70.34433820322323]
  This paper proposes an audio-visual multi-channel speech separation, dereverberation and recognition approach. The advantage of the additional visual modality over using audio only is demonstrated on two neural dereverberation approaches. Experiments conducted on the LRS2 dataset suggest that the proposed audio-visual multi-channel speech separation, dereverberation and recognition system outperforms the baseline.
  arXiv  Detail & Related papers  (2022-04-05T04:16:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.