Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR

• URL: http://arxiv.org/abs/2603.26246v1
• Date: Fri, 27 Mar 2026 10:09:30 GMT
• Title: Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR
• Abstract summary: We study whether multimodal context from prior turns improves LLM-based ASR.<n>We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities.<n>We propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly.
• Score: 14.082117520000873
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.

Related papers

• Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving [54.66679099886108]
  Speech-LLM integration has shown promising results by leveraging extensive textual pretraining, yet its specific benefits for automatic speech recognition (ASR) remain unclear.<n>We propose Joint Speech-Text Interleaved Pretraining (JSTIP), an ASR-oriented pretraining strategy that constructs word-level and segment-level interleaved speech-text sequences.<n> Experiments on 38k hours of ASR data show consistent entity accuracy improvement compared to ASR-only and joint speech-text training baselines.
  arXiv  Detail & Related papers  (2026-07-02T05:42:01Z)
• From Awareness to Adherence: Bridging the Context Gap in Spoken Dialogue Systems via Context-Aware Decoding [55.012827008735194]
  We propose an audio-adapted Context-Aware Decoding (CAD) approach to bridge the gap between latent context awareness and active adherence.<n>By leveraging internal attention mechanisms to isolate key historical rounds, our approach contrasts output with and without this key context during inference.<n> Evaluations on the Audio MultiChallenge benchmark demonstrate significant improvements in Semantic Memory and Self Coherence subtasks.
  arXiv  Detail & Related papers  (2026-06-15T09:37:11Z)
• ATIR: Towards Audio-Text Interleaved Contextual Retrieval [63.68521448682396]
  We introduce the Audio-Text Interleaved contextual Retrieval (ATIR) task, where queries can alternate between audio and text modalities.<n>This benchmark substantially addresses the limitations of existing audio retrieval datasets in semantic retrieval.
  arXiv  Detail & Related papers  (2026-04-22T07:11:58Z)
• Beyond Transcripts: A Renewed Perspective on Audio Chaptering [66.61445564139052]
  We show that a novel audio-only architecture (AudioSeg) outperforms text-based approaches for segmenting long-form audio into coherent sections.<n>Our experiments on YTSeg reveal that AudioSeg substantially outperforms text-based approaches, pauses provide the largest acoustic gains, and MLLMs remain limited by context length and weak instruction following.
  arXiv  Detail & Related papers  (2026-02-09T18:28:10Z)
• SW-ASR: A Context-Aware Hybrid ASR Pipeline for Robust Single Word Speech Recognition [0.8921166277011348]
  Single-word Automatic Speech Recognition is a challenging task due to the lack of linguistic context.<n>This paper reviews recent deep learning approaches and proposes a modular framework for robust single-word detection.<n>We evaluate the framework on the Google Speech Commands dataset and a real-world dataset collected from telephony and messaging platforms under bandwidth-limited conditions.
  arXiv  Detail & Related papers  (2026-01-28T04:50:04Z)
• MOSS-Speech: Towards True Speech-to-Speech Models Without Text Guidance [66.74042564585942]
  MOSS-Speech is a true speech-to-speech large language model that directly understands and generates speech without relying on text guidance.<n>Our work establishes a new paradigm for expressive and efficient end-to-end speech interaction.
  arXiv  Detail & Related papers  (2025-10-01T04:32:37Z)
• Internalizing ASR with Implicit Chain of Thought for Efficient Speech-to-Speech Conversational LLM [3.6950912517562435]
  We propose a method that implicitly internalizes ASR chain of thought into a speech LLM, enhancing its native speech understanding capabilities. Our approach reduces latency and improves the model's native understanding of speech, paving the way for more efficient and natural real-time audio interactions.
  arXiv  Detail & Related papers  (2024-09-25T20:59:12Z)
• Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks [68.79880423713597]
  We introduce a method that utilizes the ASR system's lattice output instead of relying solely on the top hypothesis. Our in-context learning experiments, covering spoken question answering and intent classification, underline the LLM's resilience to noisy speech transcripts.
  arXiv  Detail & Related papers  (2024-01-05T17:58:10Z)
• Generative Context-aware Fine-tuning of Self-supervised Speech Models [54.389711404209415]
  We study the use of generative large language models (LLM) generated context information. We propose an approach to distill the generated information during fine-tuning of self-supervised speech models. We evaluate the proposed approach using the SLUE and Libri-light benchmarks for several downstream tasks: automatic speech recognition, named entity recognition, and sentiment analysis.
  arXiv  Detail & Related papers  (2023-12-15T15:46:02Z)
• AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs [27.122094554340194]
  We extend the instruction-tuned Llama-2 model with end-to-end general-purpose speech processing and reasoning abilities. The resulting end-to-end model, named AudioChatLlama, can utilize audio prompts as a replacement for text and sustain a conversation.
  arXiv  Detail & Related papers  (2023-11-12T06:56:14Z)
• Learning to Ask Conversational Questions by Optimizing Levenshtein Distance [83.53855889592734]
  We introduce a Reinforcement Iterative Sequence Editing (RISE) framework that optimize the minimum Levenshtein distance (MLD) through explicit editing actions. RISE is able to pay attention to tokens that are related to conversational characteristics. Experimental results on two benchmark datasets show that RISE significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-06-30T08:44:19Z)
• Continuous speech separation: dataset and analysis [52.10378896407332]
  In natural conversations, a speech signal is continuous, containing both overlapped and overlap-free components. This paper describes a dataset and protocols for evaluating continuous speech separation algorithms.
  arXiv  Detail & Related papers  (2020-01-30T18:01:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.