Contrastive ESA: Human Evaluation of Multiple Translations at Once

• URL: http://arxiv.org/abs/2607.26640v1
• Date: Wed, 29 Jul 2026 09:01:59 GMT
• Title: Contrastive ESA: Human Evaluation of Multiple Translations at Once
• Abstract summary: We introduce Contrastive Error Span.<n>(cESA) a protocol that presents multiple translations of the source input (text, video, audio, image).<n>In cESA, the annotator sees multiple translations of the same document, marks major and minor error spans, and then assigns a score from 0% to 100% on absolute scale.<n>We validate cESA using a large-scale human evaluation of English->Japanese translations of 12 models, demonstrating reductions in annotation time and noise compared to standard pointwise evaluation.
• Score: 45.918020796451536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current human evaluation of machine translation typically assesses single outputs in isolation, a paradigm that suffers from high annotator noise and cost. We introduce Contrastive Error Span Annotation (cESA), a protocol that presents multiple translations of the source input (text, video, audio, image). In cESA, the annotator sees multiple translations of the same document, marks major and minor error spans, and then assigns a score from 0% to 100% on absolute scale. By allowing annotators to access the shared context across multiple outputs, cESA facilitates more consistent and efficient judgments. We validate cESA using a large-scale human evaluation of English->Japanese translations of 12 models, demonstrating reductions in annotation time and noise compared to standard pointwise evaluation. Unlike existing contrastive ranking methods, cESA yields absolute quality judgments that enable simple, interpretable non-parametric model rankings without the need for post-hoc corrections.

Related papers

• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation [8.48956061615262]
  Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs)<n>We address two practical gaps: how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline.<n>We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy.
  arXiv  Detail & Related papers  (2026-05-24T07:06:34Z)
• The Translation Tax Is Not a Scalar: A Counterfactual Audit of English-Source Cue Inheritance in Chinese Multilingual Benchmarks [1.4839450468199458]
  The Translation Tax is often treated as a scalar: translated benchmarks are assumed to inflate scores by preserving English-source cues.<n>Three proxy estimators disagree: back-translation gaps are small and-fragile; cue-score calibration does not predict item-level gains.<n>We release per-cell evidence, the naturalization protocol, human QC, and a reporting checklist for translated multilingual benchmark papers.
  arXiv  Detail & Related papers  (2026-05-08T01:18:46Z)
• Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck [37.780081880731096]
  Large language models (LLMs) have become a standard for multilingual evaluation, yet they exhibit a severe systematic translationese bias.<n>We attribute this bias to spurious correlations with (i) latent manifold alignment with English and (ii) cross-lingual predictability.<n>To mitigate this bias, we propose DIBJudge, a robust fine-tuning framework that learns a minimally sufficient, judgment-critical representation.
  arXiv  Detail & Related papers  (2026-03-11T02:55:29Z)
• COMET-poly: Machine Translation Metric Grounded in Other Candidates [63.82506348745169]
  We propose two automated metrics that incorporate additional information beyond the single translation.<n> COMET-polycand uses alternative translations of the same source sentence to compare and contrast with the translation at hand.<n>We find that including a single additional translation in COMET-polycand improves the segment-level metric performance.
  arXiv  Detail & Related papers  (2025-08-25T22:55:22Z)
• CLAIR-A: Leveraging Large Language Models to Judge Audio Captions [73.51087998971418]
  evaluating machine-generated audio captions is a complex task that requires considering diverse factors.<n>We propose CLAIR-A, a simple and flexible method that leverages the zero-shot capabilities of large language models.<n>In our evaluations, CLAIR-A better predicts human judgements of quality compared to traditional metrics.
  arXiv  Detail & Related papers  (2024-09-19T17:59:52Z)
• Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation [48.080874541824436]
  We introduce Error Span. ESA, a human evaluation protocol which combines the continuous rating of DA with the high-level. error severity span marking of MQM. ESA offers faster and cheaper annotations than MQM at the same quality level, without the requirement of expensive MQM experts.
  arXiv  Detail & Related papers  (2024-06-17T14:20:47Z)
• xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection [21.116517555282314]
  xCOMET is an open-source learned metric designed to bridge the gap between machine translation evaluation approaches. It integrates both sentence-level evaluation and error span detection capabilities, exhibiting state-of-the-art performance across all types of evaluation. We also provide a robustness analysis with stress tests, and show that xCOMET is largely capable of identifying localized critical errors and hallucinations.
  arXiv  Detail & Related papers  (2023-10-16T15:03:14Z)
• Using Natural Language Explanations to Rescale Human Judgments [81.66697572357477]
  We propose a method to rescale ordinal annotations and explanations using large language models (LLMs)<n>We feed annotators' Likert ratings and corresponding explanations into an LLM and prompt it to produce a numeric score anchored in a scoring rubric.<n>Our method rescales the raw judgments without impacting agreement and brings the scores closer to human judgments grounded in the same scoring rubric.
  arXiv  Detail & Related papers  (2023-05-24T06:19:14Z)
• Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgement [57.72846454929923]
  We create a benchmark dataset, emphHJQE, where the expert translators directly annotate poorly translated words. We propose two tag correcting strategies, namely tag refinement strategy and tree-based annotation strategy, to make the TER-based artificial QE corpus closer to emphHJQE. The results show our proposed dataset is more consistent with human judgement and also confirm the effectiveness of the proposed tag correcting strategies.
  arXiv  Detail & Related papers  (2022-09-13T02:37:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.