Lost in Translation: Do LVLM Judges Generalize Across Languages?

• URL: http://arxiv.org/abs/2604.19405v1
• Date: Tue, 21 Apr 2026 12:29:10 GMT
• Title: Lost in Translation: Do LVLM Judges Generalize Across Languages?
• Abstract summary: We introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation.<n> MM-JudgeBench includes over 60K pairwise preference instances spanning 25 typologically diverse languages.<n>By evaluating 22 LVLMs, we uncover substantial cross-lingual performance variance in our proposed benchmark.
• Score: 46.119587015038746
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic evaluators such as reward models play a central role in the alignment and evaluation of large vision-language models (LVLMs). Despite their growing importance, these evaluators are almost exclusively assessed on English-centric benchmarks, leaving open the question of how well these evaluators generalize across languages. To answer this question, we introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation, which includes over 60K pairwise preference instances spanning 25 typologically diverse languages. MM-JudgeBench integrates two complementary subsets: a general vision-language preference evaluation subset extending VL-RewardBench, and a chart-centric visual-text reasoning subset derived from OpenCQA, enabling systematic analysis of reward models (i.e., LVLM judges) across diverse settings. We additionally release a multilingual training set derived from MM-RewardBench, disjoint from our evaluation data, to support domain adaptation. By evaluating 22 LVLMs (15 open-source, 7 proprietary), we uncover substantial cross-lingual performance variance in our proposed benchmark. Our analysis further shows that model size and architecture are poor predictors of multilingual robustness, and that even state-of-the-art LVLM judges exhibit inconsistent behavior across languages. Together, these findings expose fundamental limitations of current reward modeling and underscore the necessity of multilingual, multimodal benchmarks for developing reliable automated evaluators.

Related papers

• Multimodal Evaluation of Russian-language Architectures [88.00147763684451]
  We introduce Mera Multi, an open multimodal evaluation framework for Russian-spoken architectures.<n>The benchmark is instruction-based and encompasses default text, image, audio, and video modalities.<n>Mera Multi provides a replicable methodology for constructing multimodal benchmarks in typologically diverse languages.
  arXiv  Detail & Related papers  (2025-11-19T15:43:53Z)
• MENLO: From Preferences to Proficiency - Evaluating and Modeling Native-like Quality Across 47 Languages [18.278876042011383]
  We introduce MENLO, a framework that operationalizes the evaluation of native-like response quality based on audience-inspired mechanisms.<n>We create a dataset of 6,423 human-annotated prompt-response preference pairs covering four quality dimensions with high inter-annotator agreement in 47 language varieties.
  arXiv  Detail & Related papers  (2025-09-30T17:48:58Z)
• M-Prometheus: A Suite of Open Multilingual LLM Judges [65.63858815169924]
  We introduce M-Prometheus, a suite of open-weight LLM judges that can provide both direct assessment and pairwise comparison feedback on multilingual outputs.<n>M-Prometheus models outperform state-of-the-art open LLM judges on multilingual reward benchmarks spanning more than 20 languages, as well as on literary machine translation (MT) evaluation covering 4 language pairs.
  arXiv  Detail & Related papers  (2025-04-07T11:37:26Z)
• Exploring the Multilingual NLG Evaluation Abilities of LLM-Based Evaluators [38.681443695708786]
  This study provides a comprehensive analysis of the multilingual evaluation performance of 10 recent LLMs.<n>We found that excluding the reference answer from the prompt leads to better performance across various languages.<n>Most LLM-based evaluators show a higher correlation with human judgments in high-resource languages than in low-resource languages.
  arXiv  Detail & Related papers  (2025-03-06T12:04:29Z)
• MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models [3.961168847961322]
  MM-Eval is a multilingual meta-evaluation benchmark covering 18 languages and a Language Consistency subset spanning 122 languages.<n>A core attribute of MM-Eval is that, instead of merely translating existing English meta-evaluation benchmarks, it is designed with multilingual-specific challenges in mind.<n>Our results show that existing evaluators that excel in English contexts have considerable room for improvement when assessing non-English outputs.
  arXiv  Detail & Related papers  (2024-10-23T06:04:55Z)
• M-RewardBench: Evaluating Reward Models in Multilingual Settings [33.44919953094607]
  We build the first-of-its-kind multilingual RM evaluation benchmark, M-RewardBench.<n>M-RewardBench consists of 2.87k preference instances for 23 typologically diverse languages.<n>We show that the performance of RMs is improved with improved translation quality.
  arXiv  Detail & Related papers  (2024-10-20T22:09:44Z)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models [71.36392373876505]
  We introduce MMIE, a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs)<n>MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts.<n>It supports both interleaved inputs and outputs, offering a mix of multiple-choice and open-ended question formats to evaluate diverse competencies.
  arXiv  Detail & Related papers  (2024-10-14T04:15:00Z)
• MMBench: Is Your Multi-modal Model an All-around Player? [114.45702807380415]
  We propose MMBench, a benchmark for assessing the multi-modal capabilities of vision-language models. MMBench is meticulously curated with well-designed quality control schemes. MMBench incorporates multiple-choice questions in both English and Chinese versions.
  arXiv  Detail & Related papers  (2023-07-12T16:23:09Z)
• Evaluating Multiway Multilingual NMT in the Turkic Languages [11.605271847666005]
  We present an evaluation of state-of-the-art approaches to training and evaluating machine translation systems in 22 languages from the Turkic language family. We train 26 bilingual baselines as well as a multi-way neural MT (MNMT) model using the corpus and perform an extensive analysis using automatic metrics as well as human evaluations. We find that the MNMT model outperforms almost all bilingual baselines in the out-of-domain test sets and finetuning the model on a downstream task of a single pair also results in a huge performance boost.
  arXiv  Detail & Related papers  (2021-09-13T19:01:07Z)
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization [128.37244072182506]
  Cross-lingual TRansfer Evaluation of Multilinguals XTREME is a benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models.
  arXiv  Detail & Related papers  (2020-03-24T19:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.