Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy

• URL: http://arxiv.org/abs/2409.09598v2
• Date: Fri, 4 Oct 2024 16:57:08 GMT
• Title: Improving Statistical Significance in Human Evaluation of Automatic Metrics via Soft Pairwise Accuracy
• Abstract summary: We propose a new meta-metric that builds on Pairwise Accuracy (PA) We show that SPA is more stable than PA with respect to changes in the number of systems/segments used for evaluation. SPA was selected as the official system-level metric for the 2024 WMT Metrics Shared Task.
• Score: 10.487125740432681
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Selecting an automatic metric that best emulates human annotators is often non-trivial, because there is no clear definition of "best emulates." A meta-metric is required to compare the human judgments to the automatic metric scores, and metric rankings depend on the choice of meta-metric. We propose Soft Pairwise Accuracy (SPA), a new meta-metric that builds on Pairwise Accuracy (PA) but incorporates the statistical significance of both the human judgments and the metric scores. We show that SPA is more stable than PA with respect to changes in the number of systems/segments used for evaluation. We also show that PA can only assign a small set of distinct output values to metrics, and this results in many metrics being artificially assigned the exact same PA score. We demonstrate that SPA fixes this issue. Finally, we show that SPA is more discriminative than PA, producing more statistically significant comparisons between metrics. SPA was selected as the official system-level metric for the 2024 WMT Metrics Shared Task.

Related papers

• Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation [24.556508865021566]
  Prediction-powered evaluation combines limited human judgments with large-scale automatic scores to obtain data-efficient system comparisons.<n>We introduce the Prediction-Powered Saving Ratio (PPSR), a meta-metric that measures how much human annotation an automatic metric can save.<n>Overall, our new paradigm reframes automatic metrics as tools for reducing human annotation cost rather than replacing human judgment, and applies broadly to non-verifiable tasks.
  arXiv  Detail & Related papers  (2026-08-27T05:48:14Z)
• GOSPA and T-GOSPA quasi-metrics for evaluation of multi-object tracking algorithms [4.759792201055905]
  This paper introduces two quasi-metrics for performance assessment of multi-object tracking (MOT) algorithms.<n>One quasi-metric is an extension of the generalised optimal subpattern assignment (GOSPA) metric and measures the discrepancy between sets of objects.<n>The other quasi-metric is an extension of the trajectory GOSPA (T-GOSPA) metric and measures the discrepancy between sets of trajectories.
  arXiv  Detail & Related papers  (2025-07-18T07:25:41Z)
• Bayesian Prediction-Powered Inference [62.2436697657307]
  Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data. We propose a framework for PPI based on Bayesian inference that allows researchers to develop new task-appropriate PPI methods easily.
  arXiv  Detail & Related papers  (2024-05-09T18:08:58Z)
• Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies [24.26653413077486]
  Ten years ago a single metric, BLEU, governed progress in machine translation research. This paper investigates the "dynamic range" of modern metrics.
  arXiv  Detail & Related papers  (2024-01-12T18:47:40Z)
• What is the Best Automated Metric for Text to Motion Generation? [19.71712698183703]
  There is growing interest in generating skeleton-based human motions from natural language descriptions. Human evaluation is the ultimate accuracy measure for this task, and automated metrics should correlate well with human quality judgments. This paper systematically studies which metrics best align with human evaluations and proposes new metrics that align even better.
  arXiv  Detail & Related papers  (2023-09-19T01:59:54Z)
• Joint Metrics Matter: A Better Standard for Trajectory Forecasting [67.1375677218281]
  Multi-modal trajectory forecasting methods evaluate using single-agent metrics (marginal metrics) Only focusing on marginal metrics can lead to unnatural predictions, such as colliding trajectories or diverging trajectories for people who are clearly walking together as a group. We present the first comprehensive evaluation of state-of-the-art trajectory forecasting methods with respect to multi-agent metrics (joint metrics): JADE, JFDE, and collision rate.
  arXiv  Detail & Related papers  (2023-05-10T16:27:55Z)
• The Glass Ceiling of Automatic Evaluation in Natural Language Generation [60.59732704936083]
  We take a step back and analyze recent progress by comparing the body of existing automatic metrics and human metrics. Our extensive statistical analysis reveals surprising findings: automatic metrics -- old and new -- are much more similar to each other than to humans.
  arXiv  Detail & Related papers  (2022-08-31T01:13:46Z)
• Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics [64.81682222169113]
  How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations. We identify two ways in which the definition of the system-level correlation is inconsistent with how metrics are used to evaluate systems in practice.
  arXiv  Detail & Related papers  (2022-04-21T15:52:14Z)
• The statistical advantage of automatic NLG metrics at the system level [10.540821585237222]
  Statistically, humans are unbiased, high variance estimators, while metrics are biased, low variance estimators. We compare these estimators by their error in pairwise prediction (which generation system is better?) using the bootstrap. Our analysis compares the adjusted error of metrics to humans and a derived, perfect segment-level annotator, both of which are unbiased estimators dependent on the number of judgments collected.
  arXiv  Detail & Related papers  (2021-05-26T09:53:57Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details [107.2722027807328]
  We find that the default implementation of AP is neither category independent, nor does it directly reward properly calibrated detectors. We show that the default implementation produces a gameable metric, where a simple, nonsensical re-ranking policy can improve AP by a large margin. We benchmark recent advances in large-vocabulary detection and find that many reported gains do not translate to improvements under our new per-class independent evaluation.
  arXiv  Detail & Related papers  (2021-02-01T18:56:02Z)
• AMR Similarity Metrics from Principles [21.915057426589748]
  We establish criteria that enable researchers to perform a principled assessment of metrics comparing meaning representations like AMR. We propose a novel metric S$2$match that is more benevolent to only very slight meaning deviations and targets the fulfilment of all established criteria.
  arXiv  Detail & Related papers  (2020-01-29T16:19:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.