Beyond Surrogates: A Quantitative Analysis for Inter-Metric Relationships

• URL: http://arxiv.org/abs/2603.07671v1
• Date: Sun, 08 Mar 2026 15:08:06 GMT
• Title: Beyond Surrogates: A Quantitative Analysis for Inter-Metric Relationships
• Abstract summary: This paper proposes a unified theoretical framework designed to quantify the relationships between metrics.<n>We categorize metrics into different classes to facilitate a comparative analysis across different mathematical forms.<n>We provide a new perspective on identifying the structural asymmetry in regret transfer, enabling the design of evaluation systems that are theoretically guaranteed to align offline improvements with online objectives.
• Score: 91.61796429377041
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Consistency property between surrogate losses and evaluation metrics has been extensively studied to ensure that minimizing a loss leads to metric optimality. However, the direct relationship between different evaluation metrics remains significantly underexplored. This theoretical gap results in the "Metric Mismatch" frequently observed in industrial applications, where gains in offline validation metrics fail to translate into online performance. To bridge this disconnection, this paper proposes a unified theoretical framework designed to quantify the relationships between metrics. We categorize metrics into different classes to facilitate a comparative analysis across different mathematical forms and interrogates these relationships through Bayes-Optimal Set and Regret Transfer. Through this framework, we provide a new perspective on identifying the structural asymmetry in regret transfer, enabling the design of evaluation systems that are theoretically guaranteed to align offline improvements with online objectives.

Related papers

• No-rank Tensor Decomposition Using Metric Learning [0.0]
  This paper introduces a no-rank tensor decomposition framework grounded in metric learning.<n>We provide theoretical guarantees for the framework's convergence and establish bounds on its metric properties.<n>Our approach achieves superior performance with smaller training datasets compared to transformer-based methods.
  arXiv  Detail & Related papers  (2025-11-03T18:21:53Z)
• Scalable Offline Metrics for Autonomous Driving [12.613436727986652]
  Extrapolating from offline model performance to online settings remains a challenge.<n>In these settings, seemingly minor errors can compound and result in test-time infractions or collisions.<n>Based on analysis in simulation, we find an even worse correlation between offline and online settings than reported by prior studies.
  arXiv  Detail & Related papers  (2025-10-09T17:59:57Z)
• Metric Design != Metric Behavior: Improving Metric Selection for the Unbiased Evaluation of Dimensionality Reduction [10.099350224451387]
  dimensionality reduction (DR) projections are crucial for reliable visual analytics.<n>DR projections can become biased if highly correlated metrics--those measuring similar structural characteristics--are inadvertently selected.<n>We propose a novel workflow that reduces bias in the selection of evaluation metrics by clustering metrics based on their empirical correlations.
  arXiv  Detail & Related papers  (2025-07-03T01:07:02Z)
• Towards an Improved Metric for Evaluating Disentangled Representations [0.6946415403594184]
  Disentangled representation learning plays a pivotal role in making representations controllable, interpretable and transferable. Despite its significance in the domain, the quest for reliable and consistent quantitative disentanglement metric remains a major challenge. We propose a new framework for quantifying disentanglement, introducing a metric entitled emphEDI, that leverages the intuitive concept of emphexclusivity and improved factor-code relationship.
  arXiv  Detail & Related papers  (2024-10-04T00:32:59Z)
• Top-K Pairwise Ranking: Bridging the Gap Among Ranking-Based Measures for Multi-Label Classification [120.37051160567277]
  This paper proposes a novel measure named Top-K Pairwise Ranking (TKPR) A series of analyses show that TKPR is compatible with existing ranking-based measures. On the other hand, we establish a sharp generalization bound for the proposed framework based on a novel technique named data-dependent contraction.
  arXiv  Detail & Related papers  (2024-07-09T09:36:37Z)
• BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training [64.37683359609308]
  In this study, we analyze various mainstream and cutting-edge automatic metrics from the perspective of their guidance for training machine translation systems. We find that certain metrics exhibit robustness defects, such as the presence of universal adversarial translations in BLEURT and BARTScore. In-depth analysis suggests two main causes of these robustness deficits: distribution biases in the training datasets, and the tendency of the metric paradigm.
  arXiv  Detail & Related papers  (2023-07-06T16:59:30Z)
• A comprehensive theoretical framework for the optimization of neural networks classification performance with respect to weighted metrics [1.0499611180329804]
  In many contexts, customized and weighted classification scores are designed in order to evaluate the goodness of predictions carried out by neural networks. We provide a complete setting that formalizes weighted classification metrics and allows the construction of losses that drive the model to optimize these interest.
  arXiv  Detail & Related papers  (2023-05-22T20:33:29Z)
• On the Importance of Asymmetry for Siamese Representation Learning [53.86929387179092]
  Siamese networks are conceptually symmetric with two parallel encoders. We study the importance of asymmetry by explicitly distinguishing the two encoders within the network. We find the improvements from asymmetric designs generalize well to longer training schedules, multiple other frameworks and newer backbones.
  arXiv  Detail & Related papers  (2022-04-01T17:57:24Z)
• REAM$\sharp$: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation [63.46331073232526]
  We present an enhancement approach to Reference-based EvAluation Metrics for open-domain dialogue systems. A prediction model is designed to estimate the reliability of the given reference set. We show how its predicted results can be helpful to augment the reference set, and thus improve the reliability of the metric.
  arXiv  Detail & Related papers  (2021-05-30T10:04:13Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.