Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge

• URL: http://arxiv.org/abs/2606.14278v1
• Date: Fri, 12 Jun 2026 08:59:26 GMT
• Title: Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge
• Abstract summary: We propose a lightweight meta-evaluation protocol that transforms LLMBar response-pair items into English, Chinese, and Chinese-English language-switched variants.<n>A reliable judge should preserve its preference under label-accessible language transformations.<n>Across models, Chinese and language-switched presentations induce 10.7--14.4% preference flips relative to English, and all judges achieve their highest accuracy in English.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are now widely used as automatic judges for open-ended instruction-following evaluation. This practice is convenient, scalable, and often more semantically aware than reference-based metrics, but it also introduces a new reliability question: does a judge evaluate the quality of an answer, or does it also react to the language in which the comparison is presented? We propose Judge-LS, a lightweight meta-evaluation protocol that transforms LLMBar response-pair items into English, Chinese, and Chinese-English language-switched variants. A reliable judge should preserve its preference under label-preserving language transformations and should not prefer a language when two answers are translation-equivalent. We evaluate four API-accessible judges on the full 419-item LLMBar benchmark, producing 13,408 successful pairwise judgments. Across models, Chinese and language-switched presentations induce 10.7--14.4% preference flips relative to English, and all judges achieve their highest accuracy in English. However, translation-equivalent tie probes do not reveal a systematic English preference: most probes are judged as ties, and non-tie decisions more often favor Chinese. We add confidence intervals, paired significance tests, and an automatic transformation audit with a sensitivity analysis that excludes mechanically flagged high-risk variants. The experiment requires no model training, uses only API calls, and is feasible on modest local hardware.

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Lower-Resource, Higher Scores: Language Bias in LLM Evaluators [27.959960538188753]
  We conduct experiments with semantically identical instruction-response pairs across 23 languages.<n>We find that multilingual evaluators assign significantly different scores to different evaluation languages.
  arXiv  Detail & Related papers  (2026-07-16T01:53:09Z)
• Cross-Lingual Stability of LLM Judges Under Controlled Generation: Evidence from Finno-Ugric Languages [0.22009842278462158]
  Cross-lingual evaluation of large language models (LLMs) typically conflates two sources of variance: genuine model performance differences and measurement instability.<n>We investigate evaluation reliability by holding generation conditions constant while varying target language.<n>Our findings suggest that zero-shot judge transfer is unreliable for discourse-level assessment in morphologically rich languages.
  arXiv  Detail & Related papers  (2026-02-02T16:27:32Z)
• Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning [74.00326879493225]
  We introduce textttSP3F, a framework for enhancing multilingual reasoning without textitany data in the target language(s)<n>First, we supervise fine-tune (SFT) on translated versions of English question-answer pairs to raise base model correctness.<n>Second, we perform RL with feedback from a pairwise judge in a self-play fashion, with the judge receiving the English reference response as textitprivileged information.
  arXiv  Detail & Related papers  (2026-01-26T17:46:44Z)
• Fairness or Fluency? An Investigation into Language Bias of Pairwise LLM-as-a-Judge [9.062065949101001]
  We study two types of language bias in pairwise LLM-as-a-judge.<n>For same-language judging, there exist significant performance disparities across language families, with European languages consistently outperforming African languages.<n>For inter-language judging, we observe that most models favor English answers, and that this preference is influenced more by answer language than question language.
  arXiv  Detail & Related papers  (2026-01-20T06:33:33Z)
• JP-TL-Bench: Anchored Pairwise LLM Evaluation for Bidirectional Japanese-English Translation [0.0]
  JP-TL-Bench is a benchmark designed to guide the iterative development of Japanese-English translation systems.<n>It evaluates a candidate model via reference-free, pairwise comparisons against a fixed, versioned anchor set.<n>Because each candidate is scored against the same frozen anchor set, scores are structurally stable given the same base set, judge, and aggregation code.
  arXiv  Detail & Related papers  (2026-01-01T06:09:45Z)
• JudgeBench: A Benchmark for Evaluating LLM-based Judges [61.048125269475854]
  JudgeBench is a benchmark for evaluating LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding.<n>Our comprehensive evaluation on a collection of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench poses a significantly greater challenge than previous benchmarks.
  arXiv  Detail & Related papers  (2024-10-16T17:58:19Z)
• CLAIR-A: Leveraging Large Language Models to Judge Audio Captions [73.51087998971418]
  evaluating machine-generated audio captions is a complex task that requires considering diverse factors.<n>We propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models.<n>In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics.
  arXiv  Detail & Related papers  (2024-09-19T17:59:52Z)
• Understanding and Mitigating Language Confusion in LLMs [76.96033035093204]
  We evaluate 15 typologically diverse languages with existing and newly-created English and multilingual prompts.<n>We find that Llama Instruct and Mistral models exhibit high degrees of language confusion.<n>We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning.
  arXiv  Detail & Related papers  (2024-06-28T17:03:51Z)
• XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization [98.61159823343036]
  We present the Word-in-Context dataset (WiC) for assessing the ability to correctly model distinct meanings of a word. We put forward a large multilingual benchmark, XL-WiC, featuring gold standards in 12 new languages. Experimental results show that even when no tagged instances are available for a target language, models trained solely on the English data can attain competitive performance.
  arXiv  Detail & Related papers  (2020-10-13T15:32:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.