Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models

• URL: http://arxiv.org/abs/2609.00727v1
• Date: Tue, 01 Sep 2026 05:03:33 GMT
• Title: Heard but Not Heeded: Paralinguistic Information Encoding and Loss in Audio-Language Models
• Abstract summary: We present a mechanistic analysis of paralinguistic information in four open source models, Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5- Omni-7B, and Chroma-4B.<n>We combine centered kernel alignment, linear probing with leave one speaker out evaluation, open ended tone prediction, and a content prosody leakage metric to trace how style information moves from the audio encoder to the final output.
• Score: 58.79935033105039
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Audio language models are designed to understand speech, yet it remains unclear whether they capture how something is said beyond what is said. We present a mechanistic analysis of paralinguistic information in four open source models, Whisper-large-v2, Qwen2-Audio-7B Instruct, Qwen2.5-Omni-7B, and Chroma-4B, using the Expresso dataset with controlled speaking styles. We combine centered kernel alignment, linear probing with leave one speaker out evaluation, open ended tone prediction, and a content prosody leakage metric to trace how style information moves from the audio encoder to the final output. All models strongly encode speaking style in the late encoder, that is, the top third of the audio encoder's layers, but this information is consistently degraded before reaching the output. The projector reshapes representation geometry without removing information, while decoders differ in how much style they preserve depending on architecture and training objective. At the output level, models fall into two behaviors. Some are content driven, where predictions depend mainly on text. Others are acoustic driven, where predictions vary with speaking style. The leakage metric quantifies this difference, and qualitative results confirm it. Overall, we identify a gap between what models encode and what they use, highlighting a key limitation in current audio language models.

Related papers

• Testing chatbots on the creation of encoders for audio conditioned image generation [0.0]
  We explore whether state-of-the-art conversational agents can design effective audio encoders to replace the CLIP text encoder.<n>We created a public demo so everyone could study and try out these audio encoders.
  arXiv  Detail & Related papers  (2025-09-09T18:57:10Z)
• From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data [55.2480439325792]
  Audio-aware large language models (ALLMs) have recently made great strides in understanding and processing audio inputs.<n>These models are typically adapted from text-based large language models (LLMs) through additional training on audio-related tasks.<n>We propose a data generation framework that produces contrastive-like training data, designed to enhance ALLMs' ability to differentiate between present and absent sounds.
  arXiv  Detail & Related papers  (2025-05-26T16:08:41Z)
• Exploring the Role of Audio in Video Captioning [59.679122191706426]
  We present an audio-visual framework, which aims to fully exploit the potential of the audio modality for captioning. We propose new local-global fusion mechanisms to improve information exchange across audio and video.
  arXiv  Detail & Related papers  (2023-06-21T20:54:52Z)
• Inter-connection: Effective Connection between Pre-trained Encoder and Decoder for Speech Translation [10.103202030679844]
  We propose an inter-connection mechanism that aggregates the information from each layer of the speech pre-trained model. This mechanism increased BLEU by approximately 2 points in en-de, en-ja, and en-zh by increasing parameters by 2K when the speech pre-trained model was frozen.
  arXiv  Detail & Related papers  (2023-05-26T13:01:29Z)
• Autodecompose: A generative self-supervised model for semantic decomposition [1.5990720051907859]
  Autodecompose is a self-supervised generative model that decomposes data into two semantically independent properties. We apply Autodecompose to audio signals to encode sound source (human voice) and content. We show that Autodecompose is robust against overfitting even when a large model is pre-trained on a small dataset.
  arXiv  Detail & Related papers  (2023-02-06T21:18:09Z)
• Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Data [145.95460945321253]
  We introduce two pre-training tasks for the encoder-decoder network using acoustic units, i.e., pseudo codes. The proposed Speech2C can relatively reduce the word error rate (WER) by 19.2% over the method without decoder pre-training.
  arXiv  Detail & Related papers  (2022-03-31T15:33:56Z)
• What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure [64.54208910952651]
  We compare audio transformer models Mockingjay and wave2vec2.0. We probe the audio models' understanding of textual surface, syntax, and semantic features. We do this over exhaustive settings for native, non-native, synthetic, read and spontaneous speech datasets.
  arXiv  Detail & Related papers  (2021-01-02T06:29:12Z)
• Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge [26.114011076658237]
  We propose two neural models to tackle the problem of learning discrete representations of speech. The first model is a type of vector-quantized variational autoencoder (VQ-VAE) The second model combines vector quantization with contrastive predictive coding (VQ-CPC) We evaluate the models on English and Indonesian data for the ZeroSpeech 2020 challenge.
  arXiv  Detail & Related papers  (2020-05-19T13:06:17Z)
• Unsupervised Audiovisual Synthesis via Exemplar Autoencoders [59.13989658692953]
  We present an unsupervised approach that converts the input speech of any individual into audiovisual streams of potentially-infinitely many output speakers. We use Exemplar Autoencoders to learn the voice, stylistic prosody, and visual appearance of a specific target speech exemplar.
  arXiv  Detail & Related papers  (2020-01-13T18:56:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.