Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning

• URL: http://arxiv.org/abs/2603.06505v1
• Date: Fri, 06 Mar 2026 17:37:06 GMT
• Title: Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning
• Abstract summary: We introduce a context-aware multilingual ASR framework that supports diverse languages and accents.<n>Our approach combines a frozen speech encoder and a decoder-only language model via a lightweight projection module.<n>We show that contextual input consistently improves recognition quality.
• Score: 5.770962296305264
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Automatic speech recognition (ASR) has benefited from advances in pretrained speech and language models, yet most systems remain constrained to monolingual settings and short, isolated utterances. While recent efforts in context-aware ASR show promise, two key challenges persist: limited multilingual support and the absence of principled alignment between speech and contextual representations. In this paper, we introduce a context-aware multilingual ASR framework that supports diverse languages and accents while preserving the modularity of pretrained models. Our approach combines a frozen speech encoder and a decoder-only language model via a lightweight projection module, allowing structured context prompts, including dialogue history and biasing words, to guide transcription. To improve interaction between speech and context, we employ a contrastive learning objective that aligns their representations in a shared embedding space. Evaluations on over 1,500 hours of real-world conversational speech across 11 languages and 5 English dialects show that contextual input consistently improves recognition quality. Contrastive alignment provides additional gains when applied to different context types, with an overall performance gain of over 5%. These results highlight the importance of both contextual modeling and cross-modal alignment in multilingual ASR.

Related papers

• MOSS-TTSD: Text to Spoken Dialogue Generation [62.04179716555789]
  We present MOSS-TTSD, a spoken dialogue synthesis model for expressive, multi-party conversational speech across multiple languages.<n>With enhanced long-context modeling, MOSS-TTSD generates long-form spoken conversations from dialogue scripts with explicit speaker tags.<n>The model supports various mainstream languages, including English and Chinese, and is adapted to several long-form scenarios.
  arXiv  Detail & Related papers  (2026-03-20T08:23:31Z)
• PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs [58.2469845374385]
  We introduce Progressive Alignment Representation Training (PART)<n>PART is a multi-stage and multi-task framework that separates within-language from cross-language alignment.<n>Experiments on CommonVoice 15, Fleurs, Wenetspeech, and CoVoST2 show that PART surpasses conventional approaches.
  arXiv  Detail & Related papers  (2025-09-24T03:54:14Z)
• Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR [23.285609467633865]
  This paper introduces the integration of language-specific bi-directional context into a speech large language model (SLLM) to improve multilingual continuous conversational automatic speech recognition (ASR)<n>We propose a character-level contextual masking strategy during training, which randomly removes portions of the context to enhance robustness and better emulate the flawed transcriptions that may occur during inference.
  arXiv  Detail & Related papers  (2025-06-16T12:03:23Z)
• What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study [58.55905182336196]
  Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation.<n>We investigate the role of speech tokenizer designs in LLM-centric SLMs, augmented by speech heads and speaker modeling.<n>We introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens.
  arXiv  Detail & Related papers  (2025-06-14T15:26:31Z)
• In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties [24.74769794165231]
  We introduce a scalable framework that allows for in-context learning (ICL) in Phi-4 Multimodal.<n>We find that as few as 12 example utterances at inference time reduce word error rates by a relative 19.7%.<n>Overall, we find that our novel ICL adaptation scheme reveals a similar performance profile to human listeners.
  arXiv  Detail & Related papers  (2025-05-20T20:20:37Z)
• VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction [114.35537839800372]
  Speech plays a crucial role in multimodal dialogue systems, and implementing high-performance in both vision and speech tasks remains a significant challenge.<n>We propose a carefully designed multi-stage training methodology that progressively trains LLM to understand both visual and speech information.<n>Our approach not only preserves strong vision-language capacity, but also enables efficient speech-to-speech dialogue capabilities without separate ASR and TTS modules.
  arXiv  Detail & Related papers  (2025-01-03T18:59:52Z)
• Channel-aware Decoupling Network for Multi-turn Dialogue Comprehension [81.47133615169203]
  We propose compositional learning for holistic interaction across utterances beyond the sequential contextualization from PrLMs. We employ domain-adaptive training strategies to help the model adapt to the dialogue domains. Experimental results show that our method substantially boosts the strong PrLM baselines in four public benchmark datasets.
  arXiv  Detail & Related papers  (2023-01-10T13:18:25Z)
• SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation [11.552745999302905]
  We propose the SAMU-XLSR: Semantically-Aligned Multimodal Utterance-level Cross-Lingual Speech Representation learning framework. We combine state-of-the-art multilingual acoustic frame-level speech representation learning model XLS-R with the Language Agnostic BERT Sentence Embedding (LaBSE) model to create an utterance-level multimodal multilingual speech encoder SAMU-XLSR.
  arXiv  Detail & Related papers  (2022-05-17T08:58:48Z)
• MAESTRO: Matched Speech Text Representations through Modality Matching [35.566604806335626]
  Maestro is a self-supervised training method to unify representations learnt from speech and text modalities. We establish a new state-of-the-art (SOTA) on VoxPopuli multilingual ASR with a 11% relative reduction in Word Error Rate (WER) We establish a new state-of-the-art (SOTA) on CoVoST 2 with an improvement of 2.8 BLEU averaged over 21 languages.
  arXiv  Detail & Related papers  (2022-04-07T12:48:16Z)
• VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion [54.29557210925752]
  One-shot voice conversion can be effectively achieved by speech representation disentanglement. We employ vector quantization (VQ) for content encoding and introduce mutual information (MI) as the correlation metric during training. Experimental results reflect the superiority of the proposed method in learning effective disentangled speech representations.
  arXiv  Detail & Related papers  (2021-06-18T13:50:38Z)
• Bridging the Modality Gap for Speech-to-Text Translation [57.47099674461832]
  End-to-end speech translation aims to translate speech in one language into text in another language via an end-to-end way. Most existing methods employ an encoder-decoder structure with a single encoder to learn acoustic representation and semantic information simultaneously. We propose a Speech-to-Text Adaptation for Speech Translation model which aims to improve the end-to-end model performance by bridging the modality gap between speech and text.
  arXiv  Detail & Related papers  (2020-10-28T12:33:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.