Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition

• URL: http://arxiv.org/abs/2603.18557v1
• Date: Thu, 19 Mar 2026 07:11:31 GMT
• Title: Cross-Lingual LLM-Judge Transfer via Evaluation Decomposition
• Abstract summary: We introduce a decomposition-based evaluation framework built around a Universal Criteria Set (UCS)<n>UCS consists of a shared, language-agnostic set of evaluation dimensions, producing an interpretable intermediate representation.<n>Experiments on multiple faithfulness tasks across languages and model backbones demonstrate consistent improvements without requiring target-language annotations.
• Score: 13.066970846043576
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models are increasingly deployed across diverse real-world applications, extending automated evaluation beyond English has become a critical challenge. Existing evaluation approaches are predominantly English-focused, and adapting them to other languages is hindered by the scarcity and cost of human-annotated judgments in most languages. We introduce a decomposition-based evaluation framework built around a Universal Criteria Set (UCS). UCS consists of a shared, language-agnostic set of evaluation dimensions, producing an interpretable intermediate representation that supports cross-lingual transfer with minimal supervision. Experiments on multiple faithfulness tasks across languages and model backbones demonstrate consistent improvements over strong baselines without requiring target-language annotations.

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• When Languages Disagree: Self-Evolving Multilingual LLM Judges [12.308063172324003]
  We show that multilingual inconsistency can provide complementary evaluation signals.<n>We propose SEMJ, a self-evolving multilingual judge that leverages cross-lingual inconsistency for iterative refinement.
  arXiv  Detail & Related papers  (2026-06-06T10:36:24Z)
• Multilingual Refusal Alignment for Safer Large Language Models [53.64286756804503]
  We investigate whether single-language alignment transfers cross-lingually, how language consistency is preserved during training, and the resulting trade-offs with general knowledge capabilities.<n>We introduce RefusEU, a novel refusal alignment dataset covering 12 European languages, including a dedicated test set for evaluating current state-of-the-art models.<n>Our controlled Direct Preference Optimization (DPO) experiments provide two key insights: aligning models exclusively in English is insufficient to ensure cross-lingual safety, even for the same harm categories, whereas training on multilingual datasets can improve safety without degrading general performance, as measured by the Global MMLU benchmark.
  arXiv  Detail & Related papers  (2026-04-24T09:29:14Z)
• All Languages Matter: Understanding and Mitigating Language Bias in Multilingual RAG [87.74635133954975]
  We show that current Multilingual Retrieval-Augmented Generation (mRAG) systems suffer from a language bias during reranking.<n>We propose textittextbfLanguage-textbfAgnostic textbfUtility-driven textbfReranker textbfAlignment (LAURA), which aligns multilingual evidence ranking with downstream generative utility.
  arXiv  Detail & Related papers  (2026-04-22T05:33:06Z)
• Lost in Translation: Do LVLM Judges Generalize Across Languages? [46.119587015038746]
  We introduce MM-JudgeBench, the first large-scale benchmark for multilingual and multimodal judge model evaluation.<n> MM-JudgeBench includes over 60K pairwise preference instances spanning 25 typologically diverse languages.<n>By evaluating 22 LVLMs, we uncover substantial cross-lingual performance variance in our proposed benchmark.
  arXiv  Detail & Related papers  (2026-04-21T12:29:10Z)
• Parallel Universes, Parallel Languages: A Comprehensive Study on LLM-based Multilingual Counterfactual Example Generation [49.2073409243885]
  Large language models (LLMs) excel at generating English counterfactuals and demonstrate multilingual proficiency.<n>We conduct automatic evaluations on both directly generated counterfactuals in the target languages and those derived via English translation across six languages.<n>We identify and categorize four main types of errors that consistently appear in the generated counterfactuals across languages.
  arXiv  Detail & Related papers  (2026-01-01T08:53:49Z)
• On the Entity-Level Alignment in Crosslingual Consistency [62.33186691736433]
  SubSub and SubInj integrate English translations of subjects into prompts across languages, leading to substantial gains in factual recall accuracy and consistency.<n>These interventions reinforce the entity representation alignment in the conceptual space through model's internal pivot-language processing.
  arXiv  Detail & Related papers  (2025-10-11T16:26:50Z)
• A Comprehensive Evaluation of Multilingual Chain-of-Thought Reasoning: Performance, Consistency, and Faithfulness Across Languages [48.68444770923683]
  We present the first comprehensive study of multilingual Chain-of-Thought (CoT) reasoning.<n>We measure language compliance, answer accuracy, and answer consistency when LRMs are prompt-hacked to think in a target language.<n>We find that the quality and effectiveness of thinking traces vary substantially depending on the prompt language.
  arXiv  Detail & Related papers  (2025-10-10T17:06:50Z)
• Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective [52.452449102961225]
  This study proposes a novel cross-linguistic perspective to investigate reasoning generalization.<n>Our findings reveal that cross-lingual transferability varies significantly across initial model, target language, and training paradigm.<n>Our study challenges the assumption that LRM reasoning mirrors human cognition, providing critical insights for the development of more language-agnostic LRMs.
  arXiv  Detail & Related papers  (2025-10-02T17:49:49Z)
• CogBench: A Large Language Model Benchmark for Multilingual Speech-Based Cognitive Impairment Assessment [23.1730341293796]
  We propose CogBench, the first benchmark designed to evaluate the cross-lingual and cross-site generalizability of large language models for speech-based cognitive impairment assessment.<n>Our results show that conventional deep learning models degrade substantially when transferred across domains.<n>Our findings offer a critical step toward building clinically useful and linguistically robust speech-based cognitive assessment tools.
  arXiv  Detail & Related papers  (2025-08-05T12:06:16Z)
• Multilingual Self-Taught Faithfulness Evaluators [11.200203292660758]
  Self-Taught Evaluators for Multilingual Faithfulness is a framework that learns exclusively from synthetic multilingual summarization data.<n>Our framework shows improvements over existing baselines, including state-of-the-art English evaluators and machine translation-based approaches.
  arXiv  Detail & Related papers  (2025-07-28T12:01:59Z)
• AdaMCoT: Rethinking Cross-Lingual Factual Reasoning through Adaptive Multilingual Chain-of-Thought [40.16140566668239]
  We introduce AdaMCOT, a framework that enhances multilingual factual reasoning.<n>AdaMCOT dynamically routing thought processes in intermediary "thinking languages" before generating target-language responses.<n>Our evaluation demonstrates substantial improvements in both factual reasoning quality and cross-lingual consistency.
  arXiv  Detail & Related papers  (2025-01-27T15:48:57Z)
• Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs [36.30321941154582]
  Hercule is a cross-lingual evaluation model that learns to assign scores to responses based on easily available reference answers in English.<n>This study is the first comprehensive examination of cross-lingual evaluation using LLMs, presenting a scalable and effective approach for multilingual assessment.
  arXiv  Detail & Related papers  (2024-10-17T09:45:32Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization [128.37244072182506]
  Cross-lingual TRansfer Evaluation of Multilinguals XTREME is a benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models.
  arXiv  Detail & Related papers  (2020-03-24T19:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.