Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluation

• URL: http://arxiv.org/abs/2604.04532v1
• Date: Mon, 06 Apr 2026 08:54:16 GMT
• Title: Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluation
• Abstract summary: We localize the Agent-as-a-Judge prompt stack to five typologically diverse languages (English, Arabic, Turkish, Chinese, Hindi)<n>We evaluate 55 DevAI development tasks across three developer-agent frameworks and six judge backbones, totaling 4950 judge runs.<n>GPT-4o achieves the highest satisfaction in English (44.72%), while Gemini leads in Arabic (51.72%, $p0.001$ vs. GPT-4o and Hindi)
• Score: 0.8253953000831505
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluation language is typically treated as a fixed English default in agentic code benchmarks, yet we show that changing the judge's language can invert backbone rankings. We localize the Agent-as-a-Judge prompt stack to five typologically diverse languages (English, Arabic, Turkish, Chinese, Hindi) and evaluate 55 DevAI development tasks across three developer-agent frameworks and six judge backbones, totaling 4950 judge runs. The central finding is that backbone and language interact: GPT-4o achieves the highest satisfaction in English (44.72\%), while Gemini leads in Arabic (51.72\%, $p<0.001$ vs.\ GPT-4o) and Hindi (53.22\%). No single backbone dominates across all languages, and inter-backbone agreement on individual requirement judgments is modest (Fleiss' $κ\leq 0.231$). A controlled ablation further shows that localizing judge-side instructions, not just benchmark content, can be decisive: Hindi satisfaction drops from 42.8\% to 23.2\% under partial localization. These results indicate that language should be treated as an explicit evaluation variable in agentic benchmarks. Full requirement-level judgments and runtime statistics are released for reproducibility.

Related papers

• EviSI: An Evidence-Based Evaluation Agent for Simultaneous Interpreting [15.392286631815763]
  EviSI is a large language model evaluation agent combining Multidimensional Quality Metrics (MQM) with criteria developed with professional interpreters.<n>On human-rated English to Chinese and Chinese to English data, EviSI recovers the aggregate English to Chinese human system ranking.<n>A multilingual extension to five directions without human ratings retains the dimensions and scoring rule, showing positive system ranking correlations with COMET throughout.
  arXiv  Detail & Related papers  (2026-09-08T03:00:11Z)
• Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator [0.8253953000831505]
  LLM judges produce evaluator-backbone rankings depending on the prompt language.<n>Top-ranked backbone alternates across English, Arabic, Chinese, Hindi, Japanese, Spanish, Turkish, and Swahili.<n>The multilingual judge score decomposes additively into task difficulty, backbone skill, and a language-backbone interaction term.
  arXiv  Detail & Related papers  (2026-08-23T14:18:25Z)
• Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge [0.0]
  We propose a lightweight meta-evaluation protocol that transforms LLMBar response-pair items into English, Chinese, and Chinese-English language-switched variants.<n>A reliable judge should preserve its preference under label-accessible language transformations.<n>Across models, Chinese and language-switched presentations induce 10.7--14.4% preference flips relative to English, and all judges achieve their highest accuracy in English.
  arXiv  Detail & Related papers  (2026-06-12T08:59:26Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• Benchmarking Commercial ASR Systems on Code-Switching Speech: Arabic, Persian, and German [0.34797121357690153]
  We present a benchmark evaluating five commercial ASR providers across four language pairs.<n>We evaluate on both WER and BERTScore, showing that while both metrics agree on the ordinal ranking of systems, WER inflates the magnitude of quality gaps.<n>ElevenLabs Scribe v2 achieves the lowest WER (13.2% overall) and leads on BERTScore (0.936 overall)
  arXiv  Detail & Related papers  (2026-05-18T19:50:44Z)
• Extending Beacon to Hindi: Cultural Adaptation Drives Cross-Lingual Sycophancy [0.0]
  Sycophancy is the tendency of language models to prioritize agreement with user preferences over principled reasoning.<n>We extend the Beacon single-turn forced-choice sycophancy diagnostic to Hindi through a controlled three-condition design.<n>Across all models, sycophancy rates are consistently higher for culturally adapted Hindi prompts than for English.
  arXiv  Detail & Related papers  (2026-01-19T12:03:52Z)
• INDIC DIALECT: A Multi Task Benchmark to Evaluate and Translate in Indian Language Dialects [10.663878830823043]
  In India, despite Hindi being the third most spoken language globally (over 600 million speakers), its numerous dialects remain underrepresented.<n>We introduce INDIC-DIALECT, a human-curated parallel corpus of 13k sentence pairs spanning 11 dialects and 2 languages: Hindi and Odia.
  arXiv  Detail & Related papers  (2026-01-15T13:40:27Z)
• One Instruction Does Not Fit All: How Well Do Embeddings Align Personas and Instructions in Low-Resource Indian Languages? [1.071318785217926]
  We present a benchmark spanning 12 Indian languages and four evaluation tasks.<n>E5-Large-Instruct achieves the highest Recall@1 of 27.4% on monolingual retrieval and 20.7% on cross-lingual transfer.<n>For classification, LaBSE attains 75.3% AUROC with strong calibration.
  arXiv  Detail & Related papers  (2026-01-15T09:10:14Z)
• HinTel-AlignBench: A Framework and Benchmark for Hindi-Telugu with English-Aligned Samples [3.3715057550177145]
  We present a scalable framework to evaluate Vision-Language Models (VLMs) in Indian languages and compare it with performance in English.<n>Using the framework, we generate HinTel-AlignBench, a benchmark that draws from diverse sources in Hindi and Telugu with English-aligned samples.<n>We find a regression in performance for tasks in English versus in Indian languages for 4 out of 5 tasks across all the models, with an average regression of 8.3 points in Hindi and 5.5 points for Telugu.
  arXiv  Detail & Related papers  (2025-11-19T07:11:00Z)
• A Multi-Language Object-Oriented Programming Benchmark for Large Language Models [61.267115598083315]
  A survey of 35 existing benchmarks uncovers three major imbalances.<n>85.7% focus on a single programming language.<n>94.3% target only function-level or statement-level tasks.<n>Over 80% include fewer than ten test cases on average.
  arXiv  Detail & Related papers  (2025-09-30T11:30:08Z)
• SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluation [52.186343500576214]
  We introduce SEAHORSE, a dataset for multilingual, multifaceted summarization evaluation. SEAHORSE consists of 96K summaries with human ratings along 6 dimensions of text quality. We show that metrics trained with SEAHORSE achieve strong performance on the out-of-domain meta-evaluation benchmarks TRUE and mFACE.
  arXiv  Detail & Related papers  (2023-05-22T16:25:07Z)
• No Language Left Behind: Scaling Human-Centered Machine Translation [69.28110770760506]
  We create datasets and models aimed at narrowing the performance gap between low and high-resource languages. We propose multiple architectural and training improvements to counteract overfitting while training on thousands of tasks. Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art.
  arXiv  Detail & Related papers  (2022-07-11T07:33:36Z)
• MIA 2022 Shared Task: Evaluating Cross-lingual Open-Retrieval Question Answering for 16 Diverse Languages [54.002969723086075]
  We evaluate cross-lingual open-retrieval question answering systems in 16 typologically diverse languages. The best system leveraging iteratively mined diverse negative examples achieves 32.2 F1, outperforming our baseline by 4.5 points. The second best system uses entity-aware contextualized representations for document retrieval, and achieves significant improvements in Tamil (20.8 F1), whereas most of the other systems yield nearly zero scores.
  arXiv  Detail & Related papers  (2022-07-02T06:54:10Z)
• CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark [144.05723617401674]
  General-purpose language intelligence evaluation has been a longstanding goal for natural language processing. We argue that for general-purpose language intelligence evaluation, the benchmark itself needs to be comprehensive and systematic. We propose CUGE, a Chinese Language Understanding and Generation Evaluation benchmark with the following features.
  arXiv  Detail & Related papers  (2021-12-27T11:08:58Z)
• On Cross-Lingual Retrieval with Multilingual Text Encoders [51.60862829942932]
  We study the suitability of state-of-the-art multilingual encoders for cross-lingual document and sentence retrieval tasks. We benchmark their performance in unsupervised ad-hoc sentence- and document-level CLIR experiments. We evaluate multilingual encoders fine-tuned in a supervised fashion (i.e., we learn to rank) on English relevance data in a series of zero-shot language and domain transfer CLIR experiments.
  arXiv  Detail & Related papers  (2021-12-21T08:10:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.