When Languages Disagree: Self-Evolving Multilingual LLM Judges

• URL: http://arxiv.org/abs/2606.08092v1
• Date: Sat, 06 Jun 2026 10:36:24 GMT
• Title: When Languages Disagree: Self-Evolving Multilingual LLM Judges
• Abstract summary: We show that multilingual inconsistency can provide complementary evaluation signals.<n>We propose SEMJ, a self-evolving multilingual judge that leverages cross-lingual inconsistency for iterative refinement.
• Score: 12.308063172324003
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multilingual LLM-as-a-judge is widely used to evaluate model outputs across languages, but suffers from cross-lingual inconsistency (Fu and Liu, 2025). Existing methods typically treat this inconsistency as noise and mitigate it through voting or aggregation. In this work, we instead show that multilingual inconsistency can provide complementary evaluation signals. Our oracle analysis finds that sampling judgments across languages yields a higher performance upper bound than single-language judging, indicating that different languages potentially include complementary judgments. Motivated by this finding, we propose SEMJ, a self-evolving multilingual judge that leverages cross-lingual inconsistency for iterative refinement. SEMJ constructs multilingual variants of each input, collects independent judgments and rationales, and feeds inconsistent outputs back for self-reflection and re-evaluation. Experiments on multiple benchmarks show that SEMJ consistently outperforms voting and reflection baselines in both accuracy and cross-lingual consistency. Further analysis shows that inconsistency triggers useful re-evaluation, which improves judgment quality.

Related papers

• Apples to Apples? Towards Comparable Crosslingual Language Model Evaluation [4.651750987298774]
  We examine crosslingual evaluation approaches using controlled monolingual language models trained on parallel data.<n>Our results show that several widely used normalized metrics introduce crosslinguistic biases rooted in tokenization, encoding, and orthographic differences.<n>In contrast, sentence-level negative log-likelihood computed over semantically equivalent sequences provides more meaningful and consistent crosslingual comparisons.
  arXiv  Detail & Related papers  (2026-08-25T19:36:15Z)
• Lower-Resource, Higher Scores: Language Bias in LLM Evaluators [27.959960538188753]
  We conduct experiments with semantically identical instruction-response pairs across 23 languages.<n>We find that multilingual evaluators assign significantly different scores to different evaluation languages.
  arXiv  Detail & Related papers  (2026-07-16T01:53:09Z)
• Disentangling Speaker and Language Effects in Cross-Lingual Speaker Verification for Iberian Languages [5.756864120253599]
  Cross-lingual speaker verification (SV) systems exhibit performance degradation when enrollment and test utterances are spoken in different languages.<n>We introduce a bilingual same-speaker evaluation set for five Iberian languages, enabling analysis of cross-lingual SV under constant speaker identity.
  arXiv  Detail & Related papers  (2026-07-01T16:44:19Z)
• All Languages Matter: Understanding and Mitigating Language Bias in Multilingual RAG [87.74635133954975]
  We show that current Multilingual Retrieval-Augmented Generation (mRAG) systems suffer from a language bias during reranking.<n>We propose textittextbfLanguage-textbfAgnostic textbfUtility-driven textbfReranker textbfAlignment (LAURA), which aligns multilingual evidence ranking with downstream generative utility.
  arXiv  Detail & Related papers  (2026-04-22T05:33:06Z)
• Lost in Translation: Do LVLM Judges Generalize Across Languages? [46.119587015038746]
  We introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation.<n> MM-JudgeBench includes over 60K pairwise preference instances spanning 25 typologically diverse languages.<n>By evaluating 22 LVLMs, we uncover substantial cross-lingual performance variance in our proposed benchmark.
  arXiv  Detail & Related papers  (2026-04-21T12:29:10Z)
• XQ-MEval: A Dataset with Cross-lingual Parallel Quality for Benchmarking Translation Metrics [64.77152900881724]
  We propose XQ-MEval, a semi-automatically built dataset covering nine translation directions.<n>We inject MQM-defined errors into gold translations automatically, filter them by native speakers for reliability, and merge errors to generate pseudo translations with controllable quality.<n>Using XQ-MEval, our experiments on nine representative metrics reveal the inconsistency between averaging and human judgment.
  arXiv  Detail & Related papers  (2026-04-16T12:27:10Z)
• Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition [13.066970846043576]
  We introduce a decomposition-based evaluation framework built around a Universal Criteria Set (UCS)<n>UCS consists of a shared, language-agnostic set of evaluation dimensions, producing an interpretable intermediate representation.<n>Experiments on multiple faithfulness tasks across languages and model backbones demonstrate consistent improvements without requiring target-language annotations.
  arXiv  Detail & Related papers  (2026-03-19T07:11:31Z)
• Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge [9.062065949101001]
  We study two types of language bias in pairwise LLM-as-a-judge.<n>For same-language judging, there exist significant performance disparities across language families, with European languages consistently outperforming African languages.<n>For inter-language judging, we observe that most models favor English answers, and that this preference is influenced more by answer language than question language.
  arXiv  Detail & Related papers  (2026-01-20T06:33:33Z)
• On the Entity-Level Alignment in Crosslingual Consistency [62.33186691736433]
  SubSub and SubInj integrate English translations of subjects into prompts across languages, leading to substantial gains in factual recall accuracy and consistency.<n>These interventions reinforce the entity representation alignment in the conceptual space through model's internal pivot-language processing.
  arXiv  Detail & Related papers  (2025-10-11T16:26:50Z)
• A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages [48.68444770923683]
  We present the first comprehensive study of multilingual Chain-of-Thought (CoT) reasoning.<n>We measure language compliance, answer accuracy, and answer consistency when LRMs are prompt-hacked to think in a target language.<n>We find that the quality and effectiveness of thinking traces vary substantially depending on the prompt language.
  arXiv  Detail & Related papers  (2025-10-10T17:06:50Z)
• Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate [36.641755706551336]
  Large language models (LLMs) provide detailed and impressive responses to queries in English.<n>But are they really consistent at responding to the same query in other languages?<n>We propose a framework to evaluate LLM's cross-lingual consistency based on a simple Translate then Evaluate strategy.
  arXiv  Detail & Related papers  (2025-05-28T06:00:21Z)
• How Reliable is Multilingual LLM-as-a-Judge? [11.639184489330368]
  We evaluate five models from different model families across five diverse tasks involving 25 languages.<n>We find that consistency varies significantly across languages, with particularly poor performance in low-resource languages.<n>We propose an ensemble strategy which improves the consistency of the multilingual judge in real-world applications.
  arXiv  Detail & Related papers  (2025-05-18T02:32:35Z)
• Understanding and Mitigating Language Confusion in LLMs [76.96033035093204]
  We evaluate 15 typologically diverse languages with existing and newly-created English and multilingual prompts.<n>We find that Llama Instruct and Mistral models exhibit high degrees of language confusion.<n>We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning.
  arXiv  Detail & Related papers  (2024-06-28T17:03:51Z)
• On Cross-Lingual Retrieval with Multilingual Text Encoders [51.60862829942932]
  We study the suitability of state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks. We benchmark their performance in unsupervised ad-hoc sentence- and document-level CLIR experiments. We evaluate multilingual encoders fine-tuned in a supervised fashion (i.e., we learn to rank) on English relevance data in a series of zero-shot language and domain transfer CLIR experiments.
  arXiv  Detail & Related papers  (2021-12-21T08:10:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.