Last Translation Benchmark

• URL: http://arxiv.org/abs/2609.04173v1
• Date: Thu, 03 Sep 2026 17:54:45 GMT
• Title: Last Translation Benchmark
• Abstract summary: The Last Translation Benchmark is a collection of examples that break leading machine translation models.<n>Each example comes with handcrafted verification rules describing concrete failure cases.<n>The latest version is LTBv1, containing accepted contributions prior to September 1st 2026.
• Score: 189.61382047977074
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, standard benchmarks for machine translation are approaching saturation. Further, automatic translation metrics are unreliable, vulnerable to reward-hacking, and provide unactionable assessments. Even gold human evaluation is not problem-free, because it often lacks reproducibility, objectivity, and scalability. Overall, this prevents us from tracking objective progress in the field and identifying pathways for improvement. We introduce the Last Translation Benchmark, a collection of human-authored and peer-reviewed examples (texts, images, audio, videos) that break leading machine translation models. We also present a new evaluation approach: each example comes with handcrafted verification rules describing concrete failure cases on that example, therefore allowing reliable and actionable future evaluation. The Last Translation Benchmark is a live dataset that accepts ongoing contributions. The latest version is LTBv1, containing accepted contributions prior to September 1st 2026, with future releases planned as new data is continuously collected.

Related papers

• Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detection [1.1340133299604382]
  We re-examine SemEval-2020 Task 1, the most influential benchmark for lexical semantic change detection.<n>We show that the benchmark is affected by substantial corpus and preprocessing problems.<n>We argue the small curated target sets and limited language coverage reduce realism and increase statistical uncertainty.
  arXiv  Detail & Related papers  (2026-04-14T19:01:25Z)
• Pretraining on the Test Set Is No Longer All You Need: A Debate-Driven Approach to QA Benchmarks [2.3188831772813105]
  We propose a debate-driven evaluation paradigm that transforms any existing QA dataset into structured adversarial debates.<n>We make two main contributions: (1) an evaluation pipeline to systematically convert QA tasks into debate-based assessments, and (2) a public benchmark that demonstrates our paradigm's effectiveness on a subset of MMLU-Pro questions.
  arXiv  Detail & Related papers  (2025-07-23T17:58:14Z)
• LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction [21.553915781660905]
  LatestEval is an automatic method that leverages the most recent texts to create uncontaminated reading comprehension evaluations. It avoids data contamination by only using texts published within a recent time window, ensuring no overlap with the training corpora of pre-trained language models. Our experiments demonstrate that language models exhibit negligible memorisation behaviours on LatestEval as opposed to previous benchmarks.
  arXiv  Detail & Related papers  (2023-12-19T17:16:43Z)
• End-to-End Evaluation for Low-Latency Simultaneous Speech Translation [55.525125193856084]
  We propose the first framework to perform and evaluate the various aspects of low-latency speech translation under realistic conditions.<n>This includes the segmentation of the audio as well as the run-time of the different components.<n>We also compare different approaches to low-latency speech translation using this framework.
  arXiv  Detail & Related papers  (2023-08-07T09:06:20Z)
• On the Blind Spots of Model-Based Evaluation Metrics for Text Generation [79.01422521024834]
  We explore a useful but often neglected methodology for robustness analysis of text generation evaluation metrics. We design and synthesize a wide range of potential errors and check whether they result in a commensurate drop in the metric scores. Our experiments reveal interesting insensitivities, biases, or even loopholes in existing metrics.
  arXiv  Detail & Related papers  (2022-12-20T06:24:25Z)
• Rethink about the Word-level Quality Estimation for Machine Translation from Human Judgement [57.72846454929923]
  We create a benchmark dataset, emphHJQE, where the expert translators directly annotate poorly translated words. We propose two tag correcting strategies, namely tag refinement strategy and tree-based annotation strategy, to make the TER-based artificial QE corpus closer to emphHJQE. The results show our proposed dataset is more consistent with human judgement and also confirm the effectiveness of the proposed tag correcting strategies.
  arXiv  Detail & Related papers  (2022-09-13T02:37:12Z)
• TRUE: Re-evaluating Factual Consistency Evaluation [29.888885917330327]
  We introduce TRUE: a comprehensive study of factual consistency metrics on a standardized collection of existing texts from diverse tasks. Our standardization enables an example-level meta-evaluation protocol that is more actionable and interpretable than previously reported correlations. Across diverse state-of-the-art metrics and 11 datasets we find that large-scale NLI and question generation-and-answering-based approaches achieve strong and complementary results.
  arXiv  Detail & Related papers  (2022-04-11T10:14:35Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• Just Rank: Rethinking Evaluation with Word and Sentence Similarities [105.5541653811528]
  intrinsic evaluation for embeddings lags far behind, and there has been no significant update since the past decade. This paper first points out the problems using semantic similarity as the gold standard for word and sentence embedding evaluations. We propose a new intrinsic evaluation method called EvalRank, which shows a much stronger correlation with downstream tasks.
  arXiv  Detail & Related papers  (2022-03-05T08:40:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.