Beyond Monolingual Deep Research: Evaluating Agents and Retrievers with Cross-Lingual BrowseComp-Plus

• URL: http://arxiv.org/abs/2606.15345v2
• Date: Wed, 17 Jun 2026 13:46:43 GMT
• Title: Beyond Monolingual Deep Research: Evaluating Agents and Retrievers with Cross-Lingual BrowseComp-Plus
• Abstract summary: XBCP (Cross-lingual BrowseComp-Plus) is a controlled benchmark that preserves the English question-and-answer space of BrowseComp-Plus but varies the languages of the supporting documents.<n>XBCP instantiates two complementary settings: in the cross-lingual setting, each query is paired with evidence in a single assigned language.<n>We evaluate four deep research agents using sparse and dense multilingual retrievers, measuring answer accuracy, evidence recall, search behavior, calibration, citation fidelity, and oracle retrieval.
• Score: 35.39357302837359
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep research agents are increasingly evaluated on their ability to search for evidence, reason over retrieved sources, and produce grounded answers. Existing browsing benchmarks, however, largely assume that the user's query and the supporting evidence are written in the same language, leaving open whether agentic search systems can operate when relevant evidence appears in another language. We introduce XBCP (Cross-lingual BrowseComp-Plus), a controlled benchmark that preserves the English question-and-answer space of BrowseComp-Plus but varies the languages of the supporting documents. XBCP instantiates two complementary settings: in the cross-lingual setting, each query is paired with evidence in a single assigned language. In the multilingual setting, the full evidence corpus is distributed equally and randomly across 12 languages spanning high-resource and low-resource regimes. We evaluate four deep research agents using sparse and dense multilingual retrievers, measuring answer accuracy, evidence recall, search behavior, calibration, citation fidelity, and oracle retrieval. Results reveal substantial degradation when evidence is translated. Even strong, dense retrievers lose evidence recall, and agents become less calibrated and cite evidence less reliably. Notably, accuracy remains lower even when all gold evidence is supplied directly. These findings suggest that cross-lingual deep research exposes both retrieval failures and an independent, agent-side difficulty in integrating language-mismatched evidence.

Related papers

• XHotpotQA: A Benchmark for Cross-Lingual Knowledge Composition in Multi-Hop Question Answering [4.102826164335728]
  XHotpotQA is a benchmark for cross-lingual knowledge composition over mixed-language evidence.<n>The audited resource contains 15,661 training and 7,405 validation instances.<n>Full question-evidence mismatch is associated with 10.25 to 15.79 lower Unicode-aware answer F1 than partial alignment.
  arXiv  Detail & Related papers  (2026-08-23T15:05:04Z)
• Diagnosing Search Behavior and Failure Modes in Long-Horizon Search Agents [63.2133766267587]
  Deep search agents answer difficult information-seeking questions by iteratively issuing search queries to gather supporting evidence.<n>We study these questions through a trajectory-level diagnosis of long-horizon search agents.<n>We evaluate the evidence retrieved at each search step and separate two stages of agent behavior: what evidence an agent retrieves and how effectively it uses that evidence.
  arXiv  Detail & Related papers  (2026-08-03T08:46:48Z)
• The Multilingual Curse at the Retrieval Layer: Evidence from Amharic [49.25758237235551]
  Strong zero-shot scores on multilingual benchmarks are often taken as evidence that current encoders transfer reliably across many languages.<n>We argue that this assumption breaks down for underrepresented, morphologically rich languages.<n>We compare zero-shot multilingual retrievers, Amharic-fine-tuned multilingual retrievers, and monolingual Amharic retrievers.
  arXiv  Detail & Related papers  (2026-05-23T12:44:30Z)
• All Languages Matter: Understanding and Mitigating Language Bias in Multilingual RAG [87.74635133954975]
  We show that current Multilingual Retrieval-Augmented Generation (mRAG) systems suffer from a language bias during reranking.<n>We propose textittextbfLanguage-textbfAgnostic textbfUtility-driven textbfReranker textbfAlignment (LAURA), which aligns multilingual evidence ranking with downstream generative utility.
  arXiv  Detail & Related papers  (2026-04-22T05:33:06Z)
• Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models [18.040787413825793]
  We introduce a benchmark of 1,500 questions spanning six tasks and five evidence scenarios.<n>The benchmark separates accessible evidence from ground truth, enabling evaluation of systems that must infer missing results from incomplete literature evidence.<n>We also present Litmus (Re)Agent, a DAG-orchestrated agentic system that decomposes queries into hypotheses, retrieves evidence, and synthesises predictions through feature-aware aggregation.
  arXiv  Detail & Related papers  (2026-04-10T05:16:33Z)
• Investigating Language and Retrieval Bias in Multilingual Previously Fact-Checked Claim Detection [4.6738956348193]
  Large Language Models (LLMs) offer powerful capabilities for cross-lingual fact-checking.<n>LLMs often exhibit language bias, performing disproportionately better on high-resource languages such as English.<n>We present and inspect a novel concept - retrieval bias, when information retrieval systems tend to favor certain information over others.
  arXiv  Detail & Related papers  (2025-09-29T17:50:32Z)
• Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness [30.00463676754559]
  We introduce BordIRLines, a dataset of territorial disputes paired with retrieved Wikipedia documents, across 49 languages.<n>We evaluate the cross-lingual robustness of this RAG setting by formalizing several modes for multilingual retrieval.<n>Our experiments show that incorporating perspectives from diverse languages can in fact improve robustness.
  arXiv  Detail & Related papers  (2024-10-02T01:59:07Z)
• Give Me More Details: Improving Fact-Checking with Latent Retrieval [58.706972228039604]
  Evidence plays a crucial role in automated fact-checking. Existing fact-checking systems either assume the evidence sentences are given or use the search snippets returned by the search engine. We propose to incorporate full text from source documents as evidence and introduce two enriched datasets.
  arXiv  Detail & Related papers  (2023-05-25T15:01:19Z)
• Evaluating and Modeling Attribution for Cross-Lingual Question Answering [80.4807682093432]
  This work is the first to study attribution for cross-lingual question answering. We collect data in 5 languages to assess the attribution level of a state-of-the-art cross-lingual QA system. We find that a substantial portion of the answers is not attributable to any retrieved passages.
  arXiv  Detail & Related papers  (2023-05-23T17:57:46Z)
• Pivot Through English: Reliably Answering Multilingual Questions without Document Retrieval [4.4973334555746]
  Existing methods for open-retrieval question answering in lower resource languages (LRLs) lag significantly behind English. We formulate a task setup more realistic to available resources, that circumvents document retrieval to reliably transfer knowledge from English to lower resource languages. Within this task setup we propose Reranked Maximal Inner Product Search (RM-MIPS), akin to semantic similarity retrieval over the English training set with reranking.
  arXiv  Detail & Related papers  (2020-12-28T04:38:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.