Source-Free MT Evaluation Is Not MT Evaluation

• URL: http://arxiv.org/abs/2608.20925v1
• Date: Fri, 21 Aug 2026 09:40:51 GMT
• Title: Source-Free MT Evaluation Is Not MT Evaluation
• Abstract summary: We argue that adequacy must be judged with respect to the source.<n>We show the existing hybrid metrics are highly reliant on reference compared to source.
• Score: 31.146042453777003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reference-based metrics remain the standard choice in machine translation evaluation, partly because quality estimation methods often correlate less well with human judgments. As a result, source-free, reference-based evaluation has become the practical norm, even though it is unfaithful to the definition of translation adequacy and unfair to systems whose outputs preserve the source meaning while differing from the reference. This paper argues that adequacy must be judged with respect to the source. A reference is only one possible rendering of the source and may introduce bias, under-specification, or errors. We further argue that source-reference-hypothesis evaluation is fair only when the judge treats the reference as auxiliary evidence rather than as the primary standard. Otherwise, even source-aware evaluation can reduce adequacy to preference towards reference. We show the existing hybrid metrics are highly reliant on reference compared to source. Our argument is not that all automatic MT metrics fail to use the source. Rather, we argue that any evaluation protocol that removes the source, or allows the reference to dominate the source, is structurally incomplete for adequacy evaluation. However, existing MT papers generally prefer reference-based metrics and use QE metrics only when reference is unavailable. We therefore call for QE to be reframed as a primary approach to source-grounded adequacy evaluation, rather than as a fallback motivated by missing references. We further call for hybrid metrics whose designs explicitly prioritize source--hypothesis faithfulness while using references only as complementary evidence.

Related papers

• CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era [51.63024682584688]
  Large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications.<n>We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing.<n>Our framework significantly outperforms prior methods in both accuracy and interpretability.
  arXiv  Detail & Related papers  (2026-02-26T19:17:39Z)
• Reference-based Metrics Disprove Themselves in Question Generation [17.83616985138126]
  We find that using human-written references cannot guarantee the effectiveness of reference-based metrics. A good metric is expected to grade a human-validated question no worse than generated questions. We propose a reference-free metric consisted of multi-dimensional criteria such as naturalness, answerability, and complexity.
  arXiv  Detail & Related papers  (2024-03-18T20:47:10Z)
• SocREval: Large Language Models with the Socratic Method for Reference-Free Reasoning Evaluation [78.23119125463964]
  We develop SocREval, a novel approach for prompt design in reference-free reasoning evaluation.<n>SocREval significantly improves GPT-4's performance, surpassing existing reference-free and reference-based reasoning evaluation metrics.
  arXiv  Detail & Related papers  (2023-09-29T18:25:46Z)
• SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window [24.524282909076767]
  We present a metric named SLIDE (SLIding Document Evaluator) which operates on blocks of sentences. We find that SLIDE obtains significantly higher pairwise system accuracy than its sentence-level baseline. This suggests that source context may provide the same information as a human reference in disambiguating source ambiguities.
  arXiv  Detail & Related papers  (2023-09-16T01:30:58Z)
• Towards Multiple References Era -- Addressing Data Leakage and Limited Reference Diversity in NLG Evaluation [55.92852268168816]
  N-gram matching-based evaluation metrics, such as BLEU and chrF, are widely utilized across a range of natural language generation (NLG) tasks. Recent studies have revealed a weak correlation between these matching-based metrics and human evaluations. We propose to utilize textitmultiple references to enhance the consistency between these metrics and human evaluations.
  arXiv  Detail & Related papers  (2023-08-06T14:49:26Z)
• Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References [123.39034752499076]
  Div-Ref is a method to enhance evaluation benchmarks by enriching the number of references. We conduct experiments to empirically demonstrate that diversifying the expression of reference can significantly enhance the correlation between automatic evaluation and human evaluation.
  arXiv  Detail & Related papers  (2023-05-24T11:53:29Z)
• DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freely [29.4981129248937]
  We propose that some reference-based metrics can be effectively adapted to assess a system summary against its corresponding reference. After being repurposed reference-freely, the zero-shot BERTScore consistently outperforms its original reference-based version. It also excels in comparison to most existing reference-free metrics and closely competes with zero-shot summary evaluators based on GPT-3.5.
  arXiv  Detail & Related papers  (2022-12-20T06:01:13Z)
• On the Limitations of Reference-Free Evaluations of Generated Text [64.81682222169113]
  We show that reference-free metrics are inherently biased and limited in their ability to evaluate generated text. We argue that they should not be used to measure progress on tasks like machine translation or summarization.
  arXiv  Detail & Related papers  (2022-10-22T22:12:06Z)
• Revisiting the Evaluation Metrics of Paraphrase Generation [35.6803390044542]
  Most existing paraphrase generation models use reference-based metrics to evaluate their generated paraphrase. This paper proposes BBScore, a reference-free metric that can reflect the generated paraphrase's quality.
  arXiv  Detail & Related papers  (2022-02-17T07:18:54Z)
• REAM$\sharp$: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation [63.46331073232526]
  We present an enhancement approach to Reference-based EvAluation Metrics for open-domain dialogue systems. A prediction model is designed to estimate the reliability of the given reference set. We show how its predicted results can be helpful to augment the reference set, and thus improve the reliability of the metric.
  arXiv  Detail & Related papers  (2021-05-30T10:04:13Z)
• BLEU might be Guilty but References are not Innocent [34.817010352734]
  We study different methods to collect references and compare their value in automated evaluation. Motivated by the finding that typical references exhibit poor diversity, concentrating around translationese language, we develop a paraphrasing task. Our method yields higher correlation with human judgment not only for the submissions of WMT 2019 English to German, but also for Back-translation and APE augmented MT output.
  arXiv  Detail & Related papers  (2020-04-13T16:49:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.