Mitigating Metric Bias in Minimum Bayes Risk Decoding

• URL: http://arxiv.org/abs/2411.03524v1
• Date: Tue, 05 Nov 2024 22:01:27 GMT
• Title: Mitigating Metric Bias in Minimum Bayes Risk Decoding
• Abstract summary: Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search. MBR decoding aims to produce translations that score highly according to a specific utility metric. This makes it impossible to use the same metric for both decoding and evaluation.
• Score: 24.97933059870959
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challenge we refer to as metric bias. As MBR decoding aims to produce translations that score highly according to a specific utility metric, this very process makes it impossible to use the same metric for both decoding and evaluation, as improvements might simply be due to reward hacking rather than reflecting real quality improvements. In this work we find that compared to human ratings, neural metrics not only overestimate the quality of MBR decoding when the same metric is used as the utility metric, but they also overestimate the quality of MBR/QE decoding with other neural utility metrics as well. We also show that the metric bias issue can be mitigated by using an ensemble of utility metrics during MBR decoding: human evaluations show that MBR decoding using an ensemble of utility metrics outperforms a single utility metric.

Related papers

• Overfitting Mitigation via Singular Value Decomposition in Minimum Bayes Risk Decoding [53.9009465845701]
  We introduce SVD-MBR, which frames the pairwise utility matrix as a noisy information signal.<n> Experiments demonstrate that SVD-MBR successfully regularizes decoding, yielding substantial gains across a range of generalized metrics.
  arXiv  Detail & Related papers  (2026-09-01T12:11:29Z)
• Noisy-Channel Minimum Bayes Risk Decoding [58.87954687016989]
  Minimum Bayes Risk (MBR) decoding yields more robust and higher-quality text generation than maximum a posteriori (MAP) decoding.<n>There exists a discrepancy in the design: hypothesis selection calculates expected utility scores conditioned on given pseudo-references.<n>It is important to consider both hypothesis-to-reference and reference-to-hypothesis directional effects.
  arXiv  Detail & Related papers  (2026-07-06T15:14:13Z)
• Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation [50.83502171176548]
  State-of-the-art generative ESD methods typically decode using Maximum a Posteriori (MAP)<n>We address this issue by applying Minimum Bayes Risk (MBR) decoding to generative ESD models.
  arXiv  Detail & Related papers  (2025-12-08T13:21:44Z)
• Agreement-Constrained Probabilistic Minimum Bayes Risk Decoding [51.82883249233765]
  We propose agreement-constrained PMBR decoding, which leverages a knowledge distilled model to guide the completion of the score matrix.<n>Our AC-PMBR decoding improved approximation errors of matrix completion by up to 3 times and achieved higher translation quality compared with PMBR decoding.
  arXiv  Detail & Related papers  (2025-12-01T06:16:47Z)
• Beyond Correlation: Interpretable Evaluation of Machine Translation Metrics [46.71836180414362]
  We introduce an interpretable evaluation framework for Machine Translation (MT) metrics. Within this framework, we evaluate metrics in two scenarios that serve as proxies for the data filtering and translation re-ranking use cases. We also raise concerns regarding the reliability of manually curated data following the Direct Assessments+Scalar Quality Metrics (DA+SQM) guidelines.
  arXiv  Detail & Related papers  (2024-10-07T16:42:10Z)
• Better Instruction-Following Through Minimum Bayes Risk [48.879360919760074]
  General-purpose LLM judges capable of human-level evaluation provide a scalable and accurate way of evaluating instruction-following LLMs. One promising way of leveraging LLM judges for supervision is through Minimum Bayes Risk (MBR) decoding. MBR decoding uses a reference-based evaluator to select a high-quality output from amongst a set of candidate outputs.
  arXiv  Detail & Related papers  (2024-10-03T18:48:38Z)
• Unveiling the Power of Source: Source-based Minimum Bayes Risk Decoding for Neural Machine Translation [30.323103270892734]
  Maximum a posteriori decoding, a commonly used method for neural machine translation (NMT), aims to maximize the estimated posterior probability. Minimum Bayes Risk (MBR) decoding offers an alternative by seeking hypotheses with the highest expected utility.
  arXiv  Detail & Related papers  (2024-06-17T15:13:52Z)
• Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms [19.543681023903456]
  We formulate Minimum Bayes Risk (MBR) decoding as a matrix completion problem. We exploit this by only computing a random subset of the scores and efficiently recover the missing entries in the matrix. Our experimental results on machine translation tasks demonstrate that the proposed method requires 1/16 utility metric computations.
  arXiv  Detail & Related papers  (2024-06-05T00:54:03Z)
• Centroid-Based Efficient Minimum Bayes Risk Decoding [38.04403087991526]
  Minimum Bayes risk (MBR) decoding achieved state-of-the-art translation performance by using COMET. MBR decoding requires quadratic time since it computes the expected score between a translation hypothesis and all reference translations. Our method clusters the reference translations in the feature space, and then calculates the score using the centroids of each cluster.
  arXiv  Detail & Related papers  (2024-02-17T05:15:12Z)
• Linear-time Minimum Bayes Risk Decoding with Reference Aggregation [52.1701152610258]
  Minimum Bayes Risk (MBR) decoding is a text generation technique that has been shown to improve the quality of machine translations. It requires the pairwise calculation of a utility metric, which has quadratic complexity. We propose to approximate pairwise metric scores with scores calculated against aggregated reference representations.
  arXiv  Detail & Related papers  (2024-02-06T18:59:30Z)
• Faster Minimum Bayes Risk Decoding with Confidence-based Pruning [8.709382540743391]
  We describe an algorithm for Minimum Bayes risk (MBR) decoding which gradually grows the number of samples used to estimate the utility. Our method requires fewer samples and drastically reduces the number of calls to the utility function compared to standard MBR. We demonstrate the effectiveness of our approach in experiments on three language pairs, using chrF++ and COMET as utility/evaluation metrics.
  arXiv  Detail & Related papers  (2023-11-25T03:38:14Z)
• Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single Model [77.19693792957614]
  We propose to make neural machine translation (NMT) models quality-aware by training them to estimate the quality of their own output. We obtain quality gains similar or even superior to quality reranking approaches, but with the efficiency of single pass decoding.
  arXiv  Detail & Related papers  (2023-10-10T15:33:51Z)
• It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk [57.641436861482696]
  Minimum Bayes Risk (MBR) decoding is a method for choosing the outputs of a machine learning system based not on the output with the highest probability, but the output with the lowest risk (expected error) among multiple candidates.
  arXiv  Detail & Related papers  (2023-10-02T17:47:10Z)
• Towards Multiple References Era -- Addressing Data Leakage and Limited Reference Diversity in NLG Evaluation [55.92852268168816]
  N-gram matching-based evaluation metrics, such as BLEU and chrF, are widely utilized across a range of natural language generation (NLG) tasks. Recent studies have revealed a weak correlation between these matching-based metrics and human evaluations. We propose to utilize textitmultiple references to enhance the consistency between these metrics and human evaluations.
  arXiv  Detail & Related papers  (2023-08-06T14:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.