Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech

• URL: http://arxiv.org/abs/2603.16606v2
• Date: Wed, 18 Mar 2026 15:24:53 GMT
• Title: Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech
• Abstract summary: Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
• Score: 61.759910921200834
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Cross-lingual sentence encoders typically cover only a few hundred languages and often trade downstream quality for stronger alignment, limiting their adoption. We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-modal sentence embedding models that natively embed text, speech, code, and mathematical expressions in a single semantic space, while delivering state-of-the-art downstream performance at the scale of thousands of languages, from high-resource to extremely low-resource varieties. To reach this scale without representation collapse, we use progressive training. We first learn a strong foundational space for 200 languages with an LLM-initialized encoder-decoder, combining token-level decoding with a novel split-softmax contrastive loss and synthetic hard negatives. Building on this foundation, we expand to several thousands language varieties via a two-stage teacher-student encoder distillation framework. Finally, we demonstrate the cross-modal extensibility of this space by seamlessly mapping 177 spoken languages into it. OmniSONAR halves cross-lingual similarity search error on the 200-language FLORES dataset and reduces error by a factor of 15 on the 1,560-language BIBLE benchmark. It also enables strong translation, outperforming NLLB-3B on multilingual benchmarks and exceeding prior models (including much larger LLMs) by 15 chrF++ points on 1,560 languages into English BIBLE translation. OmniSONAR also performs strongly on MTEB and XLCoST. For speech, OmniSONAR achieves a 43% lower similarity-search error and reaches 97% of SeamlessM4T speech-to-text quality, despite being zero-shot for translation (trained only on ASR data). Finally, by training an encoder-decoder LM, Spectrum, exclusively on English text processing OmniSONAR embedding sequences, we unlock high-performance transfer to thousands of languages and speech for complex downstream tasks.

Related papers

• OmniVoice: Towards Omnilingual Zero-Shot Text-to-Speech with Diffusion Language Models [22.0296007342245]
  We present OmniVoice, a massive multilingual text-to-speech model that scales to over 600 languages.<n>At its core is a novel diffusion language model-style discrete non-autoregressive (NAR) architecture.<n>By leveraging a 581k-hour multilingual dataset curated entirely from open-source data, OmniVoice achieves the broadest language coverage to date.
  arXiv  Detail & Related papers  (2026-04-01T09:45:51Z)
• Bilingual Text-to-Motion Generation: A New Benchmark and Baselines [52.71312720094036]
  We introduce BiHumanML3D, the first bilingual text-to-motion benchmark, constructed via LLM-assisted annotation and rigorous manual correction.<n>We also propose a simple yet effective baseline, Bilingual Motion Diffusion synthesis (BiMD), featuring Cross-Lingual Alignment (CLA)<n>BiMD with CLA achieves an FID of 0.045 vs. 0.169 and R@3 of 82.8% vs. 80.8%, significantly outperforms monolingual diffusion models and translation baselines on BiHumanML3D.
  arXiv  Detail & Related papers  (2026-03-26T08:48:27Z)
• Omnilingual MT: Machine Translation for 1,600 Languages [58.66170104105936]
  We present Omnilingual Machine Translation (OMT), the first machine translation system supporting more than 1,600 languages.<n>This scale is enabled by a comprehensive data strategy that integrates large public multilingual corpora with newly created datasets.<n>OMT models improve in cross-lingual transfer, being close to solving the "understanding" part of the puzzle in MT for the 1,600 evaluated.
  arXiv  Detail & Related papers  (2026-03-17T09:43:42Z)
• Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion [42.60008616386837]
  Speech-guided Machine Translation (SMT) framework integrates speech and text as fused inputs into an MLLM to improve translation quality.<n>Core components of this framework include a text-to-speech model, responsible for generating synthetic speech, and an MLLM capable of classifying synthetic speech samples.
  arXiv  Detail & Related papers  (2026-02-25T07:19:34Z)
• Simultaneous Speech-to-Speech Translation Without Aligned Data [52.467808474293605]
  Simultaneous speech translation requires translating source speech into a target language in real-time.<n>We propose Hibiki-Zero, which eliminates the need for word-level alignments entirely.<n>Hibiki-Zero achieves state-of-the-art performance in translation accuracy, latency, voice transfer, and naturalness across five X-to-English tasks.
  arXiv  Detail & Related papers  (2026-02-11T17:41:01Z)
• Improving Language and Modality Transfer in Translation by Character-level Modeling [14.145120349133007]
  Current translation systems, despite being highly multilingual, cover only 5% of the world's languages.<n>We propose a character-based approach to improve adaptability to new languages and modalities.
  arXiv  Detail & Related papers  (2025-05-30T13:16:08Z)
• CoVoSwitch: Machine Translation of Synthetic Code-Switched Text Based on Intonation Units [0.0]
  We synthesize code-switching data by replacing intonation units detected through PSST. We evaluate the code-switching translation performance of two multilingual translation models, M2M-100 418M and NLLB-200 600M.
  arXiv  Detail & Related papers  (2024-07-19T13:26:35Z)
• Towards Robust Speech Representation Learning for Thousands of Languages [77.2890285555615]
  Self-supervised learning (SSL) has helped extend speech technologies to more languages by reducing the need for labeled data. We propose XEUS, a Cross-lingual for Universal Speech, trained on over 1 million hours of data across 4057 languages.
  arXiv  Detail & Related papers  (2024-06-30T21:40:26Z)
• SONAR: Sentence-Level Multimodal and Language-Agnostic Representations [22.744544108288274]
  We introduce SONAR, a new multilingual and multimodal fixed-size sentence embedding space. Our single text encoder, covering 200 languages, substantially outperforms existing sentence embeddings. Speech segments can be embedded in the same SONAR embedding space using language-specific speech encoders trained in a teacher-student setting on speech transcription data.
  arXiv  Detail & Related papers  (2023-08-22T14:25:15Z)
• Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning [98.60739735409243]
  Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks. We introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts.
  arXiv  Detail & Related papers  (2023-04-03T18:46:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.