Machine Learning Evaluation Metric Discrepancies across Programming Languages and Their Components: Need for Standardization

• URL: http://arxiv.org/abs/2411.12032v2
• Date: Sat, 08 Mar 2025 21:25:11 GMT
• Title: Machine Learning Evaluation Metric Discrepancies across Programming Languages and Their Components: Need for Standardization
• Abstract summary: This study evaluates metrics for tasks such as classification, regression, clustering, correlation analysis, statistical tests, segmentation, and image-to-image (I2I) translation.<n> Metrics were compared across Python libraries, R packages, and Matlab functions to assess consistency and highlight discrepancies.
• Score: 2.4125217894669184
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: This study evaluates metrics for tasks such as classification, regression, clustering, correlation analysis, statistical tests, segmentation, and image-to-image (I2I) translation. Metrics were compared across Python libraries, R packages, and Matlab functions to assess their consistency and highlight discrepancies. The findings underscore the need for a unified roadmap to standardize metrics, ensuring reliable and reproducible ML evaluations across platforms. This study examined a wide range of evaluation metrics across various tasks and found only some to be consistent across platforms, such as (i) Accuracy, Balanced Accuracy, Cohens Kappa, F-beta Score, MCC, Geometric Mean, AUC, and Log Loss in binary classification; (ii) Accuracy, Cohens Kappa, and F-beta Score in multi-class classification; (iii) MAE, MSE, RMSE, MAPE, Explained Variance, Median AE, MSLE, and Huber in regression; (iv) Davies-Bouldin Index and Calinski-Harabasz Index in clustering; (v) Pearson, Spearman, Kendall's Tau, Mutual Information, Distance Correlation, Percbend, Shepherd, and Partial Correlation in correlation analysis; (vi) Paired t-test, Chi-Square Test, ANOVA, Kruskal-Wallis Test, Shapiro-Wilk Test, Welchs t-test, and Bartlett's test in statistical tests; (vii) Accuracy, Precision, and Recall in 2D segmentation; (viii) Accuracy in 3D segmentation; (ix) MAE, MSE, RMSE, and R-Squared in 2D-I2I translation; and (x) MAE, MSE, and RMSE in 3D-I2I translation. Given observation of discrepancies in a number of metrics (e.g. precision, recall and F1 score in binary classification, WCSS in clustering, multiple statistical tests, and IoU in segmentation, amongst multiple metrics), this study concludes that ML evaluation metrics require standardization and recommends that future research use consistent metrics for different tasks to effectively compare ML techniques and solutions.

Related papers

• T2S-Metrics: Unified Library for Evaluating SPARQL Queries Generated From Natural Language [3.0216239385077572]
  We present t2s-metrics, an open-source, unified evaluation library designed specifically for SPARQL-based assessment.<n>t2s-metrics provides a broad and set of over 20 evaluation metrics, collected from the literature and practical evaluation needs.<n>We argue that t2s-metrics constitutes a necessary step toward systematic, standardized evaluation in question answering over Knowledge Graphs.
  arXiv  Detail & Related papers  (2026-04-22T08:13:41Z)
• Magnitude Matters: a Superior Class of Similarity Metrics for Holistic Semantic Understanding [1.14219428942199]
  This paper proposes and rigorously evaluating a new class of parameter-free, magnitude-aware similarity metrics.<n>I introduce two such functions, Overlap Similarity (OS) and Hyperbolic Tangent Similarity (HTS)<n>To ensure that my findings are robust and generalizable, I conducted a comprehensive evaluation using four state-of-the-art sentence embedding models.
  arXiv  Detail & Related papers  (2025-09-12T22:26:15Z)
• AllMetrics: A Unified Python Library for Standardized Metric Evaluation and Robust Data Validation in Machine Learning [2.325084918639609]
  We introduce AllMetrics, an open-source unified Python library designed to standardize metric evaluation across diverse machine learning tasks.<n>The library implements class-specific reporting for multi-class tasks through parameters to cover all use cases.<n>Various datasets from domains like healthcare, finance, and real estate were applied to our library and compared with Python, Matlab, and R components.
  arXiv  Detail & Related papers  (2025-05-21T18:36:05Z)
• Multiclass ROC [6.941573057921458]
  We provide an evaluation metric summarizing the pair-wise multi-class True Positive Rate (TPR) and False Positive Rate (FPR) An integration over those factorized vector provides a binary AUC-equivalent summary on the performance. To support our findings, we conducted extensive simulation studies and compared our method to the pair-wise averaged AUC statistics on benchmark datasets.
  arXiv  Detail & Related papers  (2024-04-19T19:25:10Z)
• We Need to Talk About Classification Evaluation Metrics in NLP [34.73017509294468]
  In Natural Language Processing (NLP) model generalizability is generally measured with standard metrics such as Accuracy, F-Measure, or AUC-ROC. The diversity of metrics, and the arbitrariness of their application suggest that there is no agreement within NLP on a single best metric to use. We demonstrate that a random-guess normalised Informedness metric is a parsimonious baseline for task performance.
  arXiv  Detail & Related papers  (2024-01-08T11:40:48Z)
• Revisiting Evaluation Metrics for Semantic Segmentation: Optimization and Evaluation of Fine-grained Intersection over Union [113.20223082664681]
  We propose the use of fine-grained mIoUs along with corresponding worst-case metrics. These fine-grained metrics offer less bias towards large objects, richer statistical information, and valuable insights into model and dataset auditing. Our benchmark study highlights the necessity of not basing evaluations on a single metric and confirms that fine-grained mIoUs reduce the bias towards large objects.
  arXiv  Detail & Related papers  (2023-10-30T03:45:15Z)
• Revisiting Long-tailed Image Classification: Survey and Benchmarks with New Evaluation Metrics [88.39382177059747]
  A corpus of metrics is designed for measuring the accuracy, robustness, and bounds of algorithms for learning with long-tailed distribution. Based on our benchmarks, we re-evaluate the performance of existing methods on CIFAR10 and CIFAR100 datasets.
  arXiv  Detail & Related papers  (2023-02-03T02:40:54Z)
• Learning to predict test effectiveness [1.4213973379473652]
  This article offers a machine learning model to predict the extent to which the test could cover a class in terms of a new metric called Coverageability. We offer a mathematical model to evaluate test effectiveness in terms of size and coverage of the test suite generated automatically for each class.
  arXiv  Detail & Related papers  (2022-08-20T07:26:59Z)
• A New Evaluation Method: Evaluation Data and Metrics for Chinese Grammar Error Correction [4.60495447017298]
  The evaluation values of the same error correction model can vary considerably under different word segmentation systems or different language models. We propose three novel evaluation metrics for CGEC in two dimensions: reference-based and reference-less.
  arXiv  Detail & Related papers  (2022-04-30T09:40:04Z)
• MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction [51.3754092853434]
  MuCGEC is a multi-reference evaluation dataset for Chinese Grammatical Error Correction (CGEC) It consists of 7,063 sentences collected from three different Chinese-as-a-Second-Language (CSL) learner sources. Each sentence has been corrected by three annotators, and their corrections are meticulously reviewed by an expert, resulting in 2.3 references per sentence.
  arXiv  Detail & Related papers  (2022-04-23T05:20:38Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.