Anatomy of the Modality Gap: Dissecting the Internal States of End-to-End Speech LLMs

• URL: http://arxiv.org/abs/2603.01502v1
• Date: Mon, 02 Mar 2026 06:21:43 GMT
• Title: Anatomy of the Modality Gap: Dissecting the Internal States of End-to-End Speech LLMs
• Abstract summary: We analyze how speech and text representations evolve layer-by-layer.<n>We find that speech representations exhibit a broad cross-layer alignment band, attributable to the redundant nature of speech.
• Score: 15.914430317382077
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in Large Speech-Language Models have significantly bridged the gap between acoustic signals and linguistic understanding. However, a persistent performance disparity remains in speech-based input tasks compared to direct text inference. In this paper, we investigate the dynamic roots of this modality gap beyond static geometric alignment, analyzing how speech and text representations evolve layer-by-layer. We evaluate four open-weight end-to-end models on SpeechMMLU and VoiceBench BBH. Using cross-layer CKA analysis with speech-text token alignment, we find that speech representations exhibit a broad cross-layer alignment band, attributable to the redundant nature of speech where semantic content spans multiple frames. We show that these alignment patterns are structurally stable across different analysis configurations. Crucially, simple statistical calibration is insufficient and can be detrimental when applied at the input layer, indicating that the modality gap is not a mere distribution shift. Overall, our results suggest that the bottleneck lies in condensing redundant speech into stable late-layer decisions, motivating future solutions that operate at the token or temporal granularity instead of feature-level matching.

Related papers

• Do Spoken Language Models Hear Speech as They Read Text? Bridging Structural Gaps Between Speech and Text [3.1208668217678355]
  Spoken Language Models generate textual responses directly from speech.<n>Existing SLMs exhibit weaker instruction-following behavior and limited generalization.<n>We propose a simple framework that decouples length mismatch from semantic alignment.
  arXiv  Detail & Related papers  (2026-08-24T07:41:28Z)
• Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions [52.45054413627452]
  TPI-Train is a dataset of 88K instances designed with speaker-aware hard negatives to enforce acoustic cue prioritization for interruption handling.<n>TPI-Bench is a comprehensive evaluation framework designed to rigorously measure the interruption-handling strategy.
  arXiv  Detail & Related papers  (2026-04-19T10:03:42Z)
• Speech Codec Probing from Semantic and Phonetic Perspectives [49.01048570474675]
  Speech tokenizers are essential for connecting speech to large language models (LLMs) in multimodal systems.<n> emerging evidence suggests that what is termed "semantic" in speech representations does not align with text-derived semantics.
  arXiv  Detail & Related papers  (2026-03-11T03:32:25Z)
• CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillation [32.72685791637924]
  We propose CORD, a unified alignment framework that performs online cross-modal self-distillation.<n>Specifically, it aligns audio-conditioned reasoning with its text-conditioned counterpart within a unified model.<n> Empirical results across multiple benchmarks demonstrate that CORD consistently enhances audio-conditioned reasoning.
  arXiv  Detail & Related papers  (2026-01-23T08:31:24Z)
• Closing the Modality Reasoning Gap for Speech Large Language Models [33.22455377292432]
  TARS is a reinforcement-learning framework that aligns text-conditioned and speech-conditioned trajectories.<n>Our approach significantly narrows the modality reasoning gap and achieves state-of-the-art performance among 7B-scale Speech LLMs.
  arXiv  Detail & Related papers  (2026-01-09T05:51:56Z)
• Closing the Gap Between Text and Speech Understanding in LLMs [28.538793793887223]
  Large Language Models can be adapted to extend their text capabilities to speech inputs.<n>These speech-adapted LLMs consistently underperform their text-based counterparts.<n>We introduce SALAD--Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation.
  arXiv  Detail & Related papers  (2025-10-15T14:57:16Z)
• Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models [12.263637152835713]
  End-to-end Large Speech Language Models (LSLMs) have demonstrated impressive conversational generation abilities.<n>We analyze both coarse- and fine-grained text and speech representations.<n>We find that representation similarity is strongly correlated with the modality gap.
  arXiv  Detail & Related papers  (2025-10-14T03:34:38Z)
• MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance [66.74042564585942]
  MOSS-Speech is a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance.<n>Our work establishes a new paradigm for expressive and efficient end-to-end speech interaction.
  arXiv  Detail & Related papers  (2025-10-01T04:32:37Z)
• What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study [58.55905182336196]
  Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation.<n>We investigate the role of speech tokenizer designs in LLM-centric SLMs, augmented by speech heads and speaker modeling.<n>We introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens.
  arXiv  Detail & Related papers  (2025-06-14T15:26:31Z)
• Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model [76.06585781346601]
  Speech language models (Speech LMs) enable end-to-end speech-text modelling within a single model.<n>The choice of speech-text jointly decoding paradigm plays a critical role in performance, efficiency, and alignment quality.
  arXiv  Detail & Related papers  (2025-06-04T23:53:49Z)
• High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models [56.00939852727501]
  Minimally-supervised speech synthesis decouples TTS by combining two types of discrete speech representations. Non-autoregressive framework enhances controllability, and duration diffusion model enables diversified prosodic expression.
  arXiv  Detail & Related papers  (2023-09-27T09:27:03Z)
• Improving Joint Speech-Text Representations Without Alignment [92.60384956736536]
  We show that joint speech-text encoders naturally achieve consistent representations across modalities by disregarding sequence length. We argue that consistency losses could forgive length differences and simply assume the best alignment.
  arXiv  Detail & Related papers  (2023-08-11T13:28:48Z)
• End-to-End Active Speaker Detection [58.7097258722291]
  We propose an end-to-end training network where feature learning and contextual predictions are jointly learned. We also introduce intertemporal graph neural network (iGNN) blocks, which split the message passing according to the main sources of context in the ASD problem. Experiments show that the aggregated features from the iGNN blocks are more suitable for ASD, resulting in state-of-the art performance.
  arXiv  Detail & Related papers  (2022-03-27T08:55:28Z)
• Bridging the Modality Gap for Speech-to-Text Translation [57.47099674461832]
  End-to-end speech translation aims to translate speech in one language into text in another language via an end-to-end way. Most existing methods employ an encoder-decoder structure with a single encoder to learn acoustic representation and semantic information simultaneously. We propose a Speech-to-Text Adaptation for Speech Translation model which aims to improve the end-to-end model performance by bridging the modality gap between speech and text.
  arXiv  Detail & Related papers  (2020-10-28T12:33:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.