Causal Tracing of Audio-Text Fusion in Large Audio Language Models

• URL: http://arxiv.org/abs/2603.13768v1
• Date: Sat, 14 Mar 2026 05:40:59 GMT
• Title: Causal Tracing of Audio-Text Fusion in Large Audio Language Models
• Abstract summary: We adapt causal tracing to investigate the internal information flow of LALMs during audio comprehension.<n>By conducting layer-wise and token-wise analyses across DeSTA, Qwen, and Voxtral, we evaluate the causal effects of individual hidden states.
• Score: 48.849764961130795
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the strong performance of large audio language models (LALMs) in various tasks, exactly how and where they integrate acoustic features with textual context remains unclear. We adapt causal tracing to investigate the internal information flow of LALMs during audio comprehension. By conducting layer-wise and token-wise analyses across DeSTA, Qwen, and Voxtral, we evaluate the causal effects of individual hidden states. Layer-wise analysis identifies different fusion strategies, from progressive integration in DeSTA to abrupt late-stage fusion in Qwen. Token-wise analysis shows that the final sequence token acts as an informational bottleneck where the network decisively retrieves relevant information from the audio. We also observe an attention-like query mechanism at intermediate token positions that triggers the model to pull task-relevant audio context. These findings provide a clear characterization of when and where multi-modal integration occurs within LALMs.

Related papers

• What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability [6.338357793019971]
  We introduce STAG, to our knowledge the first post-hoc framework for token-level spectro-temporal grounding of captions generated by audio-based MLLMs.<n>STAG estimates the temporal support for each generated token using target-token-specific vocabulary projections of the encoded audio representations.
  arXiv  Detail & Related papers  (2026-09-11T10:08:29Z)
• From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models [19.687157920670987]
  Real-world auditory interpretation requires Context-Aware Auditory Scene Understanding (CASU)<n>We introduce the CASU benchmark, which assesses whether Audio LLMs can interpret auditory scenes composed of speech, acoustic events, and background environments.<n>We propose a scalable pipeline for constructing time-accurate, semi-synthetic audio streams by composing real-world scene sounds with synthetic speech.
  arXiv  Detail & Related papers  (2026-06-24T04:42:57Z)
• ATIR: Towards Audio-Text Interleaved Contextual Retrieval [63.68521448682396]
  We introduce the Audio-Text Interleaved contextual Retrieval (ATIR) task, where queries can alternate between audio and text modalities.<n>This benchmark substantially addresses the limitations of existing audio retrieval datasets in semantic retrieval.
  arXiv  Detail & Related papers  (2026-04-22T07:11:58Z)
• Do Audio-Visual Large Language Models Really See and Hear? [66.8531220331223]
  We analyze how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs.<n>Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.
  arXiv  Detail & Related papers  (2026-04-03T00:48:49Z)
• An Evaluation of Interleaved Instruction Tuning on Semantic Reasoning Performance in an Audio MLLM [15.340075567628466]
  This work examined the impact of interleaved instruction tuning in an audio MLLM, where audio tokens are interleaved within the prompt.<n>Our findings show that while even zero-shot interleaved prompting improves performance on our reasoning tasks, a small amount of fine-tuning improves the results further.
  arXiv  Detail & Related papers  (2025-11-04T03:54:55Z)
• Speech Discrete Tokens or Continuous Features? A Comparative Analysis for Spoken Language Understanding in SpeechLLMs [59.230858581944425]
  Two dominant approaches have emerged for speech processing: discrete tokens and continuous features.<n>We compare self-supervised learning (SSL)-based discrete and continuous features under the same experimental settings.<n>Our findings reveal that continuous features generally outperform discrete tokens in various tasks.
  arXiv  Detail & Related papers  (2025-08-25T10:16:07Z)
• When Audio and Text Disagree: Revealing Text Bias in Large Audio-Language Models [18.160420407067743]
  MCR-BENCH is the first benchmark designed to evaluate how LALMs prioritize information when presented with inconsistent audio-text pairs.<n>We reveal a concerning phenomenon: when inconsistencies exist between modalities, LALMs display a significant bias toward textual input.<n>This tendency leads to substantial performance degradation in audio-centric tasks and raises important reliability concerns for real-world applications.
  arXiv  Detail & Related papers  (2025-08-21T09:58:24Z)
• Detect Any Sound: Open-Vocabulary Sound Event Detection with Multi-Modal Queries [18.147981850263708]
  We propose a query-based framework for open-vocabulary SED guided by multi-modal queries.<n>DASM formulates SED as a frame-level retrieval task, where audio features are matched against query vectors from text or audio prompts.<n>DASM effectively balances localization accuracy with generalization to novel classes, outperforming CLAP-based methods in open-vocabulary setting.
  arXiv  Detail & Related papers  (2025-07-22T08:24:01Z)
• From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data [55.2480439325792]
  Audio-aware large language models (ALLMs) have recently made great strides in understanding and processing audio inputs.<n>These models are typically adapted from text-based large language models (LLMs) through additional training on audio-related tasks.<n>We propose a data generation framework that produces contrastive-like training data, designed to enhance ALLMs' ability to differentiate between present and absent sounds.
  arXiv  Detail & Related papers  (2025-05-26T16:08:41Z)
• Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization [50.122441710500055]
  We present LoCo, a Locality-aware cross-modal Correspondence learning framework for Audio-Visual Events (DAVE)<n>LoCo applies Local Correspondence Feature (LCF) Modulation to enforce unimodal encoders to focus on modality-shared semantics.<n>We further customize Local Adaptive Cross-modal (LAC) Interaction, which dynamically adjusts attention regions in a data-driven manner.
  arXiv  Detail & Related papers  (2024-09-12T11:54:25Z)
• DeepSafety:Multi-level Audio-Text Feature Extraction and Fusion Approach for Violence Detection in Conversations [2.8038382295783943]
  The choice of words and vocal cues in conversations presents an underexplored rich source of natural language data for personal safety and crime prevention. We introduce a new information fusion approach that extracts and fuses multi-level features including verbal, vocal, and text as heterogeneous sources of information to detect the extent of violent behaviours in conversations.
  arXiv  Detail & Related papers  (2022-06-23T16:45:50Z)
• End-to-End Active Speaker Detection [58.7097258722291]
  We propose an end-to-end training network where feature learning and contextual predictions are jointly learned. We also introduce intertemporal graph neural network (iGNN) blocks, which split the message passing according to the main sources of context in the ASD problem. Experiments show that the aggregated features from the iGNN blocks are more suitable for ASD, resulting in state-of-the art performance.
  arXiv  Detail & Related papers  (2022-03-27T08:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.