Overfitting Mitigation via Singular Value Decomposition in Minimum Bayes Risk Decoding

• URL: http://arxiv.org/abs/2609.01135v1
• Date: Tue, 01 Sep 2026 12:11:29 GMT
• Title: Overfitting Mitigation via Singular Value Decomposition in Minimum Bayes Risk Decoding
• Abstract summary: We introduce SVD-MBR, which frames the pairwise utility matrix as a noisy information signal.<n> Experiments demonstrate that SVD-MBR successfully regularizes decoding, yielding substantial gains across a range of generalized metrics.
• Score: 53.9009465845701
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Minimum Bayes Risk (MBR) decoding enables high-quality text generation by selecting the hypothesis that maximizes a utility metric over sampled pseudo-references. However, it is highly susceptible to metric overfitting: it can irregularly inflate the chosen utility metric at the direct expense of other unoptimized evaluation metrics. To mitigate this, we introduce SVD-MBR, which frames the pairwise utility matrix as a noisy information signal. By computing a low-rank approximation via Singular Value Decomposition (SVD) and retaining only the top-$k$ components, we effectively decouple true consensus from metric noise. Experiments demonstrate that SVD-MBR successfully regularizes decoding, yielding substantial gains across a range of generalized metrics. Furthermore, we reveal that this denoising is metric-dependent: neural metrics encode a robust low-rank consensus ideal for SVD, whereas surface-level metrics struggle to separate signal from metric noise.

Related papers

• Noisy-Channel Minimum Bayes Risk Decoding [58.87954687016989]
  Minimum Bayes Risk (MBR) decoding yields more robust and higher-quality text generation than maximum a posteriori (MAP) decoding.<n>There exists a discrepancy in the design: hypothesis selection calculates expected utility scores conditioned on given pseudo-references.<n>It is important to consider both hypothesis-to-reference and reference-to-hypothesis directional effects.
  arXiv  Detail & Related papers  (2026-07-06T15:14:13Z)
• Robust Recommendation from Noisy Implicit Feedback: A GMM-Weighted Bayes-label Transition Matrix Framework [3.91492263903903]
  This paper proposes a Robust GMM-weighted Bayes-label Transition Matrix framework (RGBT)<n>It simultaneously ensures full sample utilization, delivers consistent estimation, and critically, achieves a significant reduction in estimation variance.
  arXiv  Detail & Related papers  (2026-05-20T05:20:33Z)
• Optimal Null-Constrained Source-Basis Sensing in a Time-Reversed Young Interferometer [0.0]
  We develop a theory of null-constrained parameter estimation in a time-reversed Young (TRY) interferometer.<n>We show how to design source patterns that enforce a true metrological null while preserving finite first-order sensitivity to the parameter.<n>These findings identify source-basis null engineering as a distinct and practically viable modality for derivative-mode sensing.
  arXiv  Detail & Related papers  (2026-04-11T18:54:36Z)
• Beyond Raw Detection Scores: Markov-Informed Calibration for Boosting Machine-Generated Text Detection [105.14032334647932]
  Machine-generated texts (MGTs) pose risks such as disinformation and phishing, highlighting the need for reliable detection.<n> Metric-based methods, which extract statistically distinguishable features of MGTs, are often more practical than complex model-based methods that are prone to overfitting.<n>We propose a Markov-informed score calibration strategy that models two relationships of context detection scores that may aid calibration.
  arXiv  Detail & Related papers  (2026-02-08T16:06:12Z)
• Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation [50.83502171176548]
  State-of-the-art generative ESD methods typically decode using Maximum a Posteriori (MAP)<n>We address this issue by applying Minimum Bayes Risk (MBR) decoding to generative ESD models.
  arXiv  Detail & Related papers  (2025-12-08T13:21:44Z)
• Mitigating Metric Bias in Minimum Bayes Risk Decoding [24.97933059870959]
  Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search. MBR decoding aims to produce translations that score highly according to a specific utility metric. This makes it impossible to use the same metric for both decoding and evaluation.
  arXiv  Detail & Related papers  (2024-11-05T22:01:27Z)
• Risk-Sensitive Diffusion: Robustly Optimizing Diffusion Models with Noisy Samples [58.68233326265417]
  Non-image data are prevalent in real applications and tend to be noisy. Risk-sensitive SDE is a type of differential equation (SDE) parameterized by the risk vector. We conduct systematic studies for both Gaussian and non-Gaussian noise distributions.
  arXiv  Detail & Related papers  (2024-02-03T08:41:51Z)
• Towards Multiple References Era -- Addressing Data Leakage and Limited Reference Diversity in NLG Evaluation [55.92852268168816]
  N-gram matching-based evaluation metrics, such as BLEU and chrF, are widely utilized across a range of natural language generation (NLG) tasks. Recent studies have revealed a weak correlation between these matching-based metrics and human evaluations. We propose to utilize textitmultiple references to enhance the consistency between these metrics and human evaluations.
  arXiv  Detail & Related papers  (2023-08-06T14:49:26Z)
• Global Convergence of Sub-gradient Method for Robust Matrix Recovery: Small Initialization, Noisy Measurements, and Over-parameterization [4.7464518249313805]
  Sub-gradient method (SubGM) is used to recover a low-rank matrix from a limited number of measurements. We show that SubGM converges to the true solution, even under arbitrarily large and arbitrarily dense noise values.
  arXiv  Detail & Related papers  (2022-02-17T17:50:04Z)
• Attentional-Biased Stochastic Gradient Descent [74.49926199036481]
  We present a provable method (named ABSGD) for addressing the data imbalance or label noise problem in deep learning. Our method is a simple modification to momentum SGD where we assign an individual importance weight to each sample in the mini-batch. ABSGD is flexible enough to combine with other robust losses without any additional cost.
  arXiv  Detail & Related papers  (2020-12-13T03:41:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.