GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation

• URL: http://arxiv.org/abs/2604.24929v1
• Date: Mon, 27 Apr 2026 19:11:21 GMT
• Title: GAIA-v2-LILT: Multilingual Adaptation of Agent Benchmark beyond Translation
• Abstract summary: Agent benchmarks remain largely English-centric, while their multilingual versions are often built with machine translation (MT) and limited post-editing.<n>We argue that, for agentic tasks, this minimal workflow can easily break benchmark validity through query-answer misalignment or culturally off-target context.<n>We propose a refined workflow for adapting English benchmarks into multiple languages with explicit functional alignment, cultural alignment, and difficulty calibration using both automated checks and human review.
• Score: 2.845609289542755
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agent benchmarks remain largely English-centric, while their multilingual versions are often built with machine translation (MT) and limited post-editing. We argue that, for agentic tasks, this minimal workflow can easily break benchmark validity through query-answer misalignment or culturally off-target context. We propose a refined workflow for adapting English benchmarks into multiple languages with explicit functional alignment, cultural alignment, and difficulty calibration using both automated checks and human review. Using this workflow, we introduce GAIA-v2-LILT, a re-audited multilingual extension of GAIA covering five non-English languages. In experiments, our workflow improves agent success rates by up to 32.7% over minimally translated versions, bringing the closest audited setting to within 3.1% of English performance while substantial gaps remain in many other cases. This indicates that a substantial share of the multilingual performance gap is benchmark-induced measurement error, motivating task-level alignment when adapting English benchmarks across languages. The data is available as part of the MAPS package at https://huggingface.co/datasets/Fujitsu-FRE/MAPS/viewer/GAIA-v2-LILT. We also release the code used in our experiments at https://github.com/lilt/gaia-v2-lilt.

Related papers

• Translating the Translator: Decomposing the Cost of English-Forced Inter-Agent Communication [3.6466632641918113]
  Forcing inter-agent communication through English reduces Exact Match accuracy by 13.0 percentage points (Spanish) up to 30.6 percentage points (Hindi) compared to native-language multi-agent execution.<n>These findings suggest a compelling case for native-language routing in agent frameworks when the source and target languages are typologically distant.
  arXiv  Detail & Related papers  (2026-09-14T05:51:02Z)
• Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation [8.48956061615262]
  Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs)<n>We address two practical gaps: how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline.<n>We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy.
  arXiv  Detail & Related papers  (2026-05-24T07:06:34Z)
• DaPT: A Dual-Path Framework for Multilingual Multi-hop Question Answering [41.923993964253405]
  We propose a novel framework for retrieving across multilingual corpora and queries.<n>DaPT generates sub-question graphs in parallel for both the source-language query and its English translation counterpart.<n>DaPT consistently yields more accurate and concise answers compared to the baselines.
  arXiv  Detail & Related papers  (2026-03-19T16:23:04Z)
• Text2Cypher Across Languages: Evaluating and Finetuning LLMs [0.17188280334580197]
  This paper investigates the performance of both foundational and finetuned LLMs on the Text2Cypher task across multiple languages.<n>We create and release a multilingual dataset by translating English questions into Spanish and Turkish.<n>Using standardized prompts and metrics, we evaluate several foundational models and observe a consistent performance pattern.
  arXiv  Detail & Related papers  (2025-06-26T16:31:10Z)
• Breaking the Script Barrier in Multilingual Pre-Trained Language Models with Transliteration-Based Post-Training Alignment [50.27950279695363]
  The transfer performance is often hindered when a low-resource target language is written in a different script than the high-resource source language. Inspired by recent work that uses transliteration to address this problem, our paper proposes a transliteration-based post-pretraining alignment (PPA) method.
  arXiv  Detail & Related papers  (2024-06-28T08:59:24Z)
• On the Calibration of Multilingual Question Answering LLMs [57.296161186129545]
  We benchmark the calibration of several multilingual Large Language Models (MLLMs) on a variety of Question Answering tasks. We study different dimensions of calibration in in-distribution, out-of-distribution, and cross-lingual transfer settings. For decoder-only LLMs such as LlaMa2, we additionally find that in-context learning improves confidence calibration on multilingual data.
  arXiv  Detail & Related papers  (2023-11-15T03:29:02Z)
• DAMP: Doubly Aligned Multilingual Parser for Task-Oriented Dialogue [44.30286950799403]
  We show that constrastive alignment pretraining improves both English performance and transfer efficiency. Our Doubly Aligned Aligned Parsing (DAMP) improves mBERT transfer performance by 3x, 6x, and 81x on the Spanglish, Hinglish and Multilingual Task Oriented Parsing benchmarks respectively.
  arXiv  Detail & Related papers  (2022-12-15T18:58:07Z)
• Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation [80.16548523140025]
  We extend the vanilla pretrain-finetune pipeline with extra code-switching restore task to bridge the gap between the pretrain and finetune stages. Our approach could narrow the cross-lingual sentence representation distance and improve low-frequency word translation with trivial computational cost.
  arXiv  Detail & Related papers  (2022-04-16T16:08:38Z)
• On Cross-Lingual Retrieval with Multilingual Text Encoders [51.60862829942932]
  We study the suitability of state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks. We benchmark their performance in unsupervised ad-hoc sentence- and document-level CLIR experiments. We evaluate multilingual encoders fine-tuned in a supervised fashion (i.e., we learn to rank) on English relevance data in a series of zero-shot language and domain transfer CLIR experiments.
  arXiv  Detail & Related papers  (2021-12-21T08:10:27Z)
• Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrieval [51.60862829942932]
  We present a systematic empirical study focused on the suitability of the state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks. For sentence-level CLIR, we demonstrate that state-of-the-art performance can be achieved. However, the peak performance is not met using the general-purpose multilingual text encoders off-the-shelf', but rather relying on their variants that have been further specialized for sentence understanding tasks.
  arXiv  Detail & Related papers  (2021-01-21T00:15:38Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization [128.37244072182506]
  Cross-lingual TRansfer Evaluation of Multilinguals XTREME is a benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models.
  arXiv  Detail & Related papers  (2020-03-24T19:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.