Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connector

• URL: http://arxiv.org/abs/2510.00671v1
• Date: Wed, 01 Oct 2025 08:58:25 GMT
• Title: Milco: Learned Sparse Retrieval Across Languages via a Multilingual Connector
• Abstract summary: Learned Sparse Retrieval (LSR) combines the efficiency of bi-encoders with the transparency of lexical matching.<n>We introduce MILCO, an LSR architecture that maps queries and documents from different languages into a shared English lexical space.
• Score: 25.65114670027799
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learned Sparse Retrieval (LSR) combines the efficiency of bi-encoders with the transparency of lexical matching, but existing approaches struggle to scale beyond English. We introduce MILCO, an LSR architecture that maps queries and documents from different languages into a shared English lexical space via a multilingual connector. MILCO is trained with a specialized two-stage regime that combines Sparse Alignment Pretraining with contrastive training to provide representation transparency and effectiveness while mitigating semantic collapse. Motivated by the observation that uncommon entities are often lost when projected into English, we propose a new LexEcho head, which enhances robustness by augmenting the English lexical representation with a source-language view obtained through a special [ECHO] token. MILCO achieves state-of-the-art multilingual and cross-lingual LSR performance, outperforming leading dense, sparse, and multi-vector baselines such as BGE-M3 and Qwen3-Embed on standard multilingual benchmarks, while supporting dynamic efficiency through post-hoc pruning. Notably, when using mass-based pruning to reduce document representations to only 30 active dimensions on average, MILCO 560M outperforms the similarly-sized Qwen3-Embed 0.6B with 1024 dimensions.

Related papers

• MultiHaluDet: Multilingual Hallucination Detection via LLM Hidden State Probing [1.3700362496838856]
  Hallucinations in Large Language Models (LLMs) represent a critical barrier to their reliable deployment.<n>We introduce MultiHaluDet, a novel three-stage stacking framework that detects multilingual hallucinations.<n>Our framework achieves state-of-the-art detection performance, reaching up to 98.55% AUROC on the English HaluEval and TriviaQA benchmarks.
  arXiv  Detail & Related papers  (2026-05-24T07:50:03Z)
• Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech [61.759910921200834]
  Cross-lingual sentence encoders typically cover only a few hundred languages.<n>We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-language sentence embedding models.
  arXiv  Detail & Related papers  (2026-03-17T14:47:35Z)
• uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data [3.364569898365253]
  We propose a lightweight and data-efficient framework for multilingual vision-language alignment.<n>Our approach requires no image-text pairs or text-text pairs and freezes both the pretrained image encoder and multilingual text encoder during training.<n>This minimal training setup enables robust multilingual alignment even for languages with limited supervision.
  arXiv  Detail & Related papers  (2025-11-17T06:34:49Z)
• Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-training [57.62126373849383]
  Cross-lingual In-context Pre-training (CrossIC-PT) is a simple and scalable approach that enhances cross-lingual transfer.<n>We construct CrossIC-PT samples by interleaving semantic-related bilingual Wikipedia documents into a single context window.<n> Experimental results demonstrate that CrossIC-PT improves multilingual performance on three models across six target languages.
  arXiv  Detail & Related papers  (2025-04-29T07:24:25Z)
• COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing [1.2242530642524063]
  COMI-LINGUA is the largest manually annotated Hindi-English code-mixed dataset.<n>It comprises 125K+ high-quality instances across five core NLP tasks.<n>Each instance is annotated by three bilingual annotators, yielding over 376K expert annotations.
  arXiv  Detail & Related papers  (2025-03-27T16:36:39Z)
• Parrot: Multilingual Visual Instruction Tuning [66.65963606552839]
  Existing methods typically align vision encoders with Multimodal Large Language Models (MLLMs) via supervised fine-tuning (SFT)<n>We propose PARROT, a novel approach that leverages textual guidance for visual token alignment at the language level.<n>We introduce the Massive Multilingual Multimodal Benchmark (MMMB), a new benchmark comprising 6 languages, 15 categories, and 12,000 questions.
  arXiv  Detail & Related papers  (2024-06-04T17:56:28Z)
• Learning Multilingual Sentence Representations with Cross-lingual Consistency Regularization [46.09132547431629]
  We introduce MuSR: a one-for-all Multilingual Sentence Representation model that supports more than 220 languages. We train a multilingual Transformer encoder, coupled with an auxiliary Transformer decoder, by adopting a multilingual NMT framework. Experimental results on multilingual similarity search and bitext mining tasks show the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-12T07:39:06Z)
• Chain-of-Dictionary Prompting Elicits Translation in Large Language Models [100.47154959254937]
  Large language models (LLMs) have shown surprisingly good performance in multilingual neural machine translation (MNMT) We present a novel method, CoD, which augments LLMs with prior knowledge with the chains of multilingual dictionaries for a subset of input words to elicit translation abilities.
  arXiv  Detail & Related papers  (2023-05-11T05:19:47Z)
• VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning [56.47303426167584]
  We propose a cross-lingual pre-trained model VECO2.0 based on contrastive learning with multi-granularity alignments. Specifically, the sequence-to-sequence alignment is induced to maximize the similarity of the parallel pairs and minimize the non-parallel pairs. token-to-token alignment is integrated to bridge the gap between synonymous tokens excavated via the thesaurus dictionary from the other unpaired tokens in a bilingual instance.
  arXiv  Detail & Related papers  (2023-04-17T12:23:41Z)
• Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning [98.60739735409243]
  Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks. We introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts.
  arXiv  Detail & Related papers  (2023-04-03T18:46:01Z)
• Exposing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders [85.80950708769923]
  We probe multilingual language models for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs. We also devise a novel method to expose this knowledge by additionally fine-tuning multilingual models. We report substantial gains on standard benchmarks.
  arXiv  Detail & Related papers  (2022-04-30T13:23:16Z)
• Learning Disentangled Semantic Representations for Zero-Shot Cross-Lingual Transfer in Multilingual Machine Reading Comprehension [40.38719019711233]
  Multilingual pre-trained models are able to zero-shot transfer knowledge from rich-resource languages to low-resource languages in machine reading comprehension (MRC) In this paper, we propose a novel multilingual MRC framework equipped with a Siamese Semantic Disentanglement Model (SSDM) to disassociate semantics from syntax in representations learned by multilingual pre-trained models.
  arXiv  Detail & Related papers  (2022-04-03T05:26:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.