MLAIRE: Multilingual Language-Aware Information Retrieval Evaluation Protocal

• URL: http://arxiv.org/abs/2605.07249v1
• Date: Fri, 08 May 2026 05:10:05 GMT
• Title: MLAIRE: Multilingual Language-Aware Information Retrieval Evaluation Protocal
• Abstract summary: MLAIRE is a Multilingual Language-Aware Information Retrieval Evaluation protocol.<n>It disentangles cross-lingual semantic retrieval from query-language preference.<n>We show that standard metrics obscure distinct behaviors.
• Score: 26.186599286240362
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multilingual Information Retrieval is increasingly important in real-world search settings, where users issue queries over mixed-language corpora. Existing evaluations mainly reward language-agnostic semantic relevance, treating relevant passages equally regardless of language. Yet retrieval utility also depends on the language of the retrieved passages: users may prefer results they can read and verify in the query language, and query--passage language mismatch can complicate downstream grounding and answer verification in Retrieval-Augmented Generation systems. To evaluate this language-aware dimension, we introduce MLAIRE, a Multilingual Language-Aware Information Retrieval Evaluation protocol that disentangles cross-lingual semantic retrieval from query-language preference. MLAIRE constructs controlled pools with parallel passages across languages, enabling measurement of semantic retrieval accuracy and query-language preference when equivalent translations are available. We propose language-aware metrics, including Language Preference Rate (LPR) and Lang-nDCG, together with a 4-way decomposition separating semantic and query-language preference failures. Evaluating 31 dense, sparse, and late-interaction retrievers, we show that standard metrics obscure distinct behaviors: semantically strong retrievers may return correct content in a non-query language, while retrievers with stronger query-language preference may retrieve less semantically relevant passages.

Related papers

• Disentangled Contrastive Learning for Zero-Shot Multilingual Dense Retrieval [18.12664959335536]
  We propose a disentangled contrastive learning(DCL) method for multilingual dense retrieval.<n>Specifically, we design disentangled optimization objectives based on hierarchical semantic alignment and language debiasing contrastive learning.<n>Our method consistently outperforms several strong baselines, demonstrating its effectiveness and generalization ability.
  arXiv  Detail & Related papers  (2026-08-03T13:13:15Z)
• LAMAR: An Open Language-Aware Multilingual Alignment Reranker [24.064060212702838]
  We show that multilingual rerankers do not consistently prioritize documents written in the same language as the query.<n>We release LAMAR, a language aware multilingual cross encoder trained to account for both semantic relevance and language coherence.
  arXiv  Detail & Related papers  (2026-07-24T07:11:42Z)
• SHIFT: Semantic Harmonization via Index-side Feature Transformation for Multilingual Information Retrieval [26.186599286240362]
  Multilingual Information Retrieval (MLIR) has emerged as a critical technology for global information access.<n>Recent multilingual dense retrieval models often exhibit a strong preference for documents in the same language as the query.<n>We propose SHIFT, a training-free method applicable in the indexing stage.
  arXiv  Detail & Related papers  (2026-06-17T08:14:51Z)
• Generative Query Expansion with Multilingual LLMs for Cross-Lingual Information Retrieval [0.19116784879310025]
  Multilingual large language models (mLLMs) have shifted query expansion from semantic augmentation with synonyms and related words to pseudo-document generation.<n>This study evaluates recent mLLMs and fine-tuned variants across several generative expansion strategies to identify factors that drive cross-lingual retrieval performance.
  arXiv  Detail & Related papers  (2025-11-24T17:18:25Z)
• Bridging Language Gaps: Advances in Cross-Lingual Information Retrieval with Multilingual LLMs [0.19116784879310025]
  Cross-lingual information retrieval (CLIR) addresses the challenge of retrieving relevant documents written in languages different from that of the original query.<n>Recent advances have shifted from translation-based methods toward embedding-based approaches.<n>This survey provides a comprehensive overview of developments from early translation-based methods to state-of-the-art embedding-driven and generative techniques.
  arXiv  Detail & Related papers  (2025-10-01T13:50:05Z)
• Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG [55.258582772528506]
  We investigate whether the mixture of different document languages impacts generation and citation in unintended ways.<n>Across eight languages and six open-weight models, we find that models preferentially cite English sources when queries are in English.<n>We find that models sometimes trade-off document relevance for language preference, indicating that citation choices are not always driven by informativeness alone.
  arXiv  Detail & Related papers  (2025-09-17T12:58:18Z)
• The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corpora [5.0908395672023055]
  Cross-lingual retrieval-augmented generation (RAG) is a critical capability for retrieving and generating answers across languages.<n>We study Arabic-English RAG in a domain-specific setting using benchmarks derived from real-world corporate datasets.<n>We propose two simple retrieval strategies that address this source of failure by enforcing equal retrieval from both languages or by translating the query.
  arXiv  Detail & Related papers  (2025-07-10T08:38:31Z)
• mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval [61.17793165194077]
  We introduce mFollowIR, a benchmark for measuring instruction-following ability in retrieval models.<n>We present results for both multilingual (XX-XX) and cross-lingual (En-XX) performance.<n>We see strong cross-lingual performance with English-based retrievers that trained using instructions, but find a notable drop in performance in the multilingual setting.
  arXiv  Detail & Related papers  (2025-01-31T16:24:46Z)
• Understanding and Mitigating Language Confusion in LLMs [76.96033035093204]
  We evaluate 15 typologically diverse languages with existing and newly-created English and multilingual prompts.<n>We find that Llama Instruct and Mistral models exhibit high degrees of language confusion.<n>We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning.
  arXiv  Detail & Related papers  (2024-06-28T17:03:51Z)
• MINERS: Multilingual Language Models as Semantic Retrievers [23.686762008696547]
  This paper introduces the MINERS, a benchmark designed to evaluate the ability of multilingual language models in semantic retrieval tasks. We create a comprehensive framework to assess the robustness of LMs in retrieving samples across over 200 diverse languages. Our results demonstrate that by solely retrieving semantically similar embeddings yields performance competitive with state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-06-11T16:26:18Z)
• On Cross-Lingual Retrieval with Multilingual Text Encoders [51.60862829942932]
  We study the suitability of state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks. We benchmark their performance in unsupervised ad-hoc sentence- and document-level CLIR experiments. We evaluate multilingual encoders fine-tuned in a supervised fashion (i.e., we learn to rank) on English relevance data in a series of zero-shot language and domain transfer CLIR experiments.
  arXiv  Detail & Related papers  (2021-12-21T08:10:27Z)
• Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations [83.27475281544868]
  We use singular vector canonical correlation analysis to study what kind of information is induced from each source. We observe that our representations embed typology and strengthen correlations with language relationships. We then take advantage of our multi-view language vector space for multilingual machine translation, where we achieve competitive overall translation accuracy.
  arXiv  Detail & Related papers  (2020-04-30T16:25:39Z)
• On the Importance of Word Order Information in Cross-lingual Sequence Labeling [80.65425412067464]
  Cross-lingual models that fit into the word order of the source language might fail to handle target languages. We investigate whether making models insensitive to the word order of the source language can improve the adaptation performance in target languages.
  arXiv  Detail & Related papers  (2020-01-30T03:35:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.