Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation

• URL: http://arxiv.org/abs/2603.12983v2
• Date: Mon, 16 Mar 2026 10:15:07 GMT
• Title: Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation
• Abstract summary: Error Span Detection is a crucial subtask in Machine Translation (MT) evaluation.<n>We propose a novel self-evolution framework based on Minimum Bayes Risk (MBR) decoding.<n>It eliminates the reliance on human annotations by leveraging an off-the-shelf LLM to generate pseudo-labels.
• Score: 4.3983707233782745
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Error Span Detection (ESD) is a crucial subtask in Machine Translation (MT) evaluation, aiming to identify the location and severity of translation errors. While fine-tuning models on human-annotated data improves ESD performance, acquiring such data is expensive and prone to inconsistencies among annotators. To address this, we propose a novel self-evolution framework based on Minimum Bayes Risk (MBR) decoding, named Iterative MBR Distillation for ESD, which eliminates the reliance on human annotations by leveraging an off-the-shelf LLM to generate pseudo-labels. Extensive experiments on the WMT Metrics Shared Task datasets demonstrate that models trained solely on these self-generated pseudo-labels outperform both unadapted base model and supervised baselines trained on human annotations at the system and span levels, while maintaining competitive sentence-level performance.

Related papers

• Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation [50.83502171176548]
  State-of-the-art generative ESD methods typically decode using Maximum a Posteriori (MAP)<n>We address this issue by applying Minimum Bayes Risk (MBR) decoding to generative ESD models.
  arXiv  Detail & Related papers  (2025-12-08T13:21:44Z)
• Multi-Agent VLMs Guided Self-Training with PNU Loss for Low-Resource Offensive Content Detection [32.68131638705225]
  We propose a self-training framework that leverages abundant unlabeled data through collaborative pseudo-labeling.<n>Our method iteratively assigns pseudo-labels to unlabeled instances with the support of Multi-Agent Vision-Language Models.<n>Experiments on benchmark datasets demonstrate that our framework substantially outperforms baselines under limited supervision.
  arXiv  Detail & Related papers  (2025-11-14T08:03:35Z)
• LLM-Assisted Semantic Guidance for Sparsely Annotated Remote Sensing Object Detection [25.9348571356454]
  LLM-assisted semantic guidance framework tailored for sparsely annotated remote sensing object detection.<n>Dense Pseudo-Label Assignment mechanism adaptively assigns pseudo-labels for both unlabeled and sparsely labeled data.<n> Adaptive Hard-Negative Reweighting Module to stabilize the supervised learning branch.
  arXiv  Detail & Related papers  (2025-09-21T08:05:43Z)
• Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimation [55.73341401764367]
  We introduce DCSQE, a novel framework for alleviating distribution shift in synthetic QE data.<n> DCSQE uses references, i.e., translation supervision signals, to guide both the generation and annotation processes.<n>Experiments demonstrate that DCSQE outperforms SOTA baselines in both supervised and unsupervised settings.
  arXiv  Detail & Related papers  (2025-02-27T10:11:53Z)
• RDBE: Reasoning Distillation-Based Evaluation Enhances Automatic Essay Scoring [0.0]
  Reasoning Distillation-Based Evaluation (RDBE) integrates interpretability to elucidate the rationale behind model scores. Our experimental results demonstrate the efficacy of RDBE across all scoring rubrics considered in the dataset.
  arXiv  Detail & Related papers  (2024-07-03T05:49:01Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing [72.14557106085284]
  slice detection models (SDM) automatically identify underperforming groups of datapoints. This paper proposes a benchmark named "Discover, Explain, improve (DEIM)" for classification NLP tasks. Our evaluation shows that Edisa can accurately select error-prone datapoints with informative semantic features.
  arXiv  Detail & Related papers  (2022-11-08T19:00:00Z)
• Unsupervised Domain Adaptive Salient Object Detection Through Uncertainty-Aware Pseudo-Label Learning [104.00026716576546]
  We propose to learn saliency from synthetic but clean labels, which naturally has higher pixel-labeling quality without the effort of manual annotations. We show that our proposed method outperforms the existing state-of-the-art deep unsupervised SOD methods on several benchmark datasets.
  arXiv  Detail & Related papers  (2022-02-26T16:03:55Z)
• Identifying Weaknesses in Machine Translation Metrics Through Minimum Bayes Risk Decoding: A Case Study for COMET [42.77140426679383]
  We show that sample-based Minimum Bayes Risk decoding can be used to explore and quantify such weaknesses. We further show that these biases cannot be fully removed by simply training on additional synthetic data.
  arXiv  Detail & Related papers  (2022-02-10T17:07:32Z)
• On the Inference Calibration of Neural Machine Translation [54.48932804996506]
  We study the correlation between calibration and translation performance and linguistic properties of miscalibration. We propose a new graduated label smoothing method that can improve both inference calibration and translation performance.
  arXiv  Detail & Related papers  (2020-05-03T02:03:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.