Ranking evaluation metrics from a group-theoretic perspective

• URL: http://arxiv.org/abs/2408.16009v1
• Date: Wed, 14 Aug 2024 09:06:58 GMT
• Title: Ranking evaluation metrics from a group-theoretic perspective
• Abstract summary: We show instances resulting in inconsistent evaluations, sources of potential mistrust in commonly used metrics. Our analysis sheds light on ranking evaluation metrics, highlighting that inconsistent evaluations should not be seen as a source of mistrust.
• Score: 5.333192842860574
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Confronted with the challenge of identifying the most suitable metric to validate the merits of newly proposed models, the decision-making process is anything but straightforward. Given that comparing rankings introduces its own set of formidable challenges and the likely absence of a universal metric applicable to all scenarios, the scenario does not get any better. Furthermore, metrics designed for specific contexts, such as for Recommender Systems, sometimes extend to other domains without a comprehensive grasp of their underlying mechanisms, resulting in unforeseen outcomes and potential misuses. Complicating matters further, distinct metrics may emphasize different aspects of rankings, frequently leading to seemingly contradictory comparisons of model results and hindering the trustworthiness of evaluations. We unveil these aspects in the domain of ranking evaluation metrics. Firstly, we show instances resulting in inconsistent evaluations, sources of potential mistrust in commonly used metrics; by quantifying the frequency of such disagreements, we prove that these are common in rankings. Afterward, we conceptualize rankings using the mathematical formalism of symmetric groups detaching from possible domains where the metrics have been created; through this approach, we can rigorously and formally establish essential mathematical properties for ranking evaluation metrics, essential for a deeper comprehension of the source of inconsistent evaluations. We conclude with a discussion, connecting our theoretical analysis to the practical applications, highlighting which properties are important in each domain where rankings are commonly evaluated. In conclusion, our analysis sheds light on ranking evaluation metrics, highlighting that inconsistent evaluations should not be seen as a source of mistrust but as the need to carefully choose how to evaluate our models in the future.

Related papers

• To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Bias [61.40435736418359]
  We introduce a unified framework that standardizes heterogeneous benchmarks to contrast isolated demographic assessments with forced-choice comparative settings.<n>We show that comparative settings act as aggressive catalysts for latent discrimination, a shift primarily driven by underspecified contexts.<n>Finally, we demonstrate that this comparative prejudice is a generalized phenomenon that scales positively with model size.
  arXiv  Detail & Related papers  (2026-06-23T13:53:50Z)
• Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews [69.66583722746904]
  We introduce Beyond Rating, a holistic evaluation framework that assesses AI reviewers across five dimensions.<n>We propose a Max-Recall strategy to accommodate valid expert disagreement and introduce a curated dataset of paper with high-confidence reviews.<n>Our proposed text-centric metrics--particularly the recall of weakness arguments--correlate strongly with rating accuracy.
  arXiv  Detail & Related papers  (2026-04-21T14:21:15Z)
• When Fairness Metrics Disagree: Evaluating the Reliability of Demographic Fairness Assessment in Machine Learning [0.0]
  We investigate the consistency of fairness evaluation by conducting a systematic multi-metric analysis of demographic bias in machine learning models.<n>Our results demonstrate that fairness assessments can vary significantly depending on the choice of metrics, leading to contradictory conclusions regarding model bias.<n>These findings highlight a critical limitation in current fairness evaluation practices and suggest that single-metric reporting is insufficient for reliable bias assessment.
  arXiv  Detail & Related papers  (2026-04-16T14:07:37Z)
• Do Metrics for Counterfactual Explanations Align with User Perception? [3.919854840656461]
  We compare algorithmic evaluation metrics with human judgments of explanation quality.<n>Our results show that correlations between algorithmic metrics and human ratings are generally weak and strongly dataset-dependent.<n>Our findings suggest that widely used counterfactual evaluation metrics fail to reflect key aspects of explanation quality as perceived by users.
  arXiv  Detail & Related papers  (2026-03-16T17:56:54Z)
• DREAM: Deep Research Evaluation with Agentic Metrics [21.555357444628044]
  We propose DREAM (Deep Research Evaluation with Agentic Metrics), a framework that makes evaluation itself agentic.<n> DREAM structures assessment through an evaluation protocol combining query-agnostic metrics with adaptive metrics generated by a tool-calling agent.<n>Controlled evaluations demonstrate DREAM is significantly more sensitive to factual and temporal decay than existing benchmarks.
  arXiv  Detail & Related papers  (2026-02-21T19:14:31Z)
• The Validity of Coreference-based Evaluations of Natural Language Understanding [3.505146496638911]
  I analyze standard coreference evaluations and show that their design often leads to non-generalizable conclusions.<n>I propose and implement a novel evaluation focused on testing systems' ability to infer the relative plausibility of events.
  arXiv  Detail & Related papers  (2026-02-18T05:49:28Z)
• Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification [9.99813918008511]
  We study prompt-dependent ranking inference under pairwise human preferences.<n>We develop a framework for decision-safe rankings with statistically valid uncertainty guarantees.
  arXiv  Detail & Related papers  (2026-02-11T18:16:24Z)
• CCE: Confidence-Consistency Evaluation for Time Series Anomaly Detection [56.302586730134806]
  We introduce Confidence-Consistency Evaluation (CCE), a novel evaluation metric.<n>CCE simultaneously measures prediction confidence and uncertainty consistency.<n>We also establish RankEval, a benchmark for comparing the ranking capabilities of various metrics.
  arXiv  Detail & Related papers  (2025-09-01T03:38:38Z)
• Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness [49.35494016290887]
  We show that equal performance across subgroups is an unreliable measure of fairness when data are representative of relevant populations but reflective of real-world disparities.<n>Our framework suggests complementing disaggregated evaluations with explicit causal assumptions and analysis to control for confounding and distribution shift.
  arXiv  Detail & Related papers  (2025-06-04T17:40:31Z)
• Algorithmic Accountability in Small Data: Sample-Size-Induced Bias Within Classification Metrics [0.0]
  We show the significance of sample-size bias in classification metrics.<n>This revelation challenges the efficacy of these metrics in assessing bias with high resolution.<n>We propose a model-agnostic assessment and correction technique.
  arXiv  Detail & Related papers  (2025-05-06T22:02:53Z)
• Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy [52.261323452286554]
  We introduce a method for contextual metric meta-evaluation by comparing the local metric accuracy of evaluation metrics. Across translation, speech recognition, and ranking tasks, we demonstrate that the local metric accuracies vary both in absolute value and relative effectiveness as we shift across evaluation contexts.
  arXiv  Detail & Related papers  (2025-03-25T16:42:25Z)
• SEOE: A Scalable and Reliable Semantic Evaluation Framework for Open Domain Event Detection [70.23196257213829]
  We propose a scalable and reliable Semantic-level Evaluation framework for Open domain Event detection. Our proposed framework first constructs a scalable evaluation benchmark that currently includes 564 event types covering 7 major domains. We then leverage large language models (LLMs) as automatic evaluation agents to compute a semantic F1-score, incorporating fine-grained definitions of semantically similar labels.
  arXiv  Detail & Related papers  (2025-03-05T09:37:05Z)
• Evaluating Step-by-step Reasoning Traces: A Survey [3.895864050325129]
  We propose a taxonomy of evaluation criteria with four top-level categories (groundedness, validity, coherence, and utility) We then categorize metrics based on their implementations, survey which metrics are used for assessing each criterion, and explore whether evaluator models can transfer across different criteria.
  arXiv  Detail & Related papers  (2025-02-17T19:58:31Z)
• Confidence Diagram of Nonparametric Ranking for Uncertainty Assessment in Large Language Models Evaluation [20.022623972491733]
  Ranking large language models (LLMs) has proven to be an effective tool to improve alignment based on the best-of-$N$ policy. We propose a new inferential framework for hypothesis testing among the ranking for language models.
  arXiv  Detail & Related papers  (2024-12-07T02:34:30Z)
• On the Evaluation Consistency of Attribution-based Explanations [42.1421504321572]
  We introduce Meta-Rank, an open platform for benchmarking attribution methods in the image domain. Our benchmark reveals three insights in attribution evaluation endeavors: 1) evaluating attribution methods under disparate settings can yield divergent performance rankings; 2) although inconsistent across numerous cases, the performance rankings exhibit remarkable consistency across distinct checkpoints along the same training trajectory; and 3) prior attempts at consistent evaluation fare no better than baselines when extended to more heterogeneous models and datasets.
  arXiv  Detail & Related papers  (2024-07-28T11:49:06Z)
• Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics [22.041561519672456]
  Large language models (LLMs) often produce unsupported or unverifiable content, known as "hallucinations" We propose a comparative evaluation framework that assesses the metric effectiveness in distinguishing citations between three-category support levels. Our results show no single metric consistently excels across all evaluations, revealing the complexity of assessing fine-grained support.
  arXiv  Detail & Related papers  (2024-06-21T15:57:24Z)
• A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice [6.091702876917282]
  Classification systems are evaluated in a countless number of papers. However, we find that evaluation practice is often nebulous. Many works use so-called'macro' metrics to rank systems but do not clearly specify what they would expect from such a metric.
  arXiv  Detail & Related papers  (2024-04-25T18:12:43Z)
• Forecast Evaluation and the Relationship of Regret and Calibration [8.28720658988688]
  We provide a general structure which subsumes many currently used evaluation metrics in a two-dimensional hierarchy.<n>The framework embeds those evaluation metrics in a large set of single-instance-based comparisons of forecasts and observations.<n>In particular, this framework sheds light on the relationship on regret-type and calibration-type evaluation metrics showing a theoretical equivalence in their ability to evaluate, but practical incomparability of the obtained scores.
  arXiv  Detail & Related papers  (2024-01-25T19:36:11Z)
• Discordance Minimization-based Imputation Algorithms for Missing Values in Rating Data [4.100928307172084]
  When multiple rating lists are combined or considered together, subjects often have missing ratings. We propose analyses on missing value patterns using six real-world data sets in various applications. We propose optimization models and algorithms that minimize the total rating discordance across rating providers.
  arXiv  Detail & Related papers  (2023-11-07T14:42:06Z)
• KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluation [69.57018875757622]
  We propose KPEval, a comprehensive evaluation framework consisting of four critical aspects: reference agreement, faithfulness, diversity, and utility. Using KPEval, we re-evaluate 23 keyphrase systems and discover that established model comparison results have blind-spots.
  arXiv  Detail & Related papers  (2023-03-27T17:45:38Z)
• Properties of Group Fairness Metrics for Rankings [4.479834103607384]
  We perform a comparative analysis of existing group fairness metrics developed in the context of fair ranking. We take an axiomatic approach whereby we design a set of thirteen properties for group fairness metrics. We demonstrate that most of these metrics only satisfy a small subset of the proposed properties.
  arXiv  Detail & Related papers  (2022-12-29T15:50:18Z)
• Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries [59.27273928454995]
  Current pre-trained models applied to summarization are prone to factual inconsistencies which misrepresent the source text or introduce extraneous information. We create a crowdsourcing evaluation framework for factual consistency using the rating-based Likert scale and ranking-based Best-Worst Scaling protocols. We find that ranking-based protocols offer a more reliable measure of summary quality across datasets, while the reliability of Likert ratings depends on the target dataset and the evaluation design.
  arXiv  Detail & Related papers  (2021-09-19T19:05:00Z)
• Estimation of Fair Ranking Metrics with Incomplete Judgments [70.37717864975387]
  We propose a sampling strategy and estimation technique for four fair ranking metrics. We formulate a robust and unbiased estimator which can operate even with very limited number of labeled items.
  arXiv  Detail & Related papers  (2021-08-11T10:57:00Z)
• REAM$\sharp$: An Enhancement Approach to Reference-based Evaluation Metrics for Open-domain Dialog Generation [63.46331073232526]
  We present an enhancement approach to Reference-based EvAluation Metrics for open-domain dialogue systems. A prediction model is designed to estimate the reliability of the given reference set. We show how its predicted results can be helpful to augment the reference set, and thus improve the reliability of the metric.
  arXiv  Detail & Related papers  (2021-05-30T10:04:13Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning [70.72363097550483]
  In this study, we focus on in-domain uncertainty for image classification. To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE)
  arXiv  Detail & Related papers  (2020-02-15T23:28:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.