Related papers: Goodness of Fit Metrics for Multi-class Predictor

Goodness of Fit Metrics for Multi-class Predictor

URL: http://arxiv.org/abs/2208.05651v1
Date: Thu, 11 Aug 2022 06:07:29 GMT
Title: Goodness of Fit Metrics for Multi-class Predictor
Authors: Uri Itai, Natan Katz
Abstract summary: Several metrics are commonly used to measure fit goodness. A leading constraint at least in emphreal world multi-class problems is imbalanced data. We suggest generalizing Matthew's correlation coefficient into multi-dimensions.
Score: 0.0
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: The multi-class prediction had gained popularity over recent years. Thus measuring fit goodness becomes a cardinal question that researchers often have to deal with. Several metrics are commonly used for this task. However, when one has to decide about the right measurement, he must consider that different use-cases impose different constraints that govern this decision. A leading constraint at least in \emph{real world} multi-class problems is imbalanced data: Multi categorical problems hardly provide symmetrical data. Hence, when we observe common KPIs (key performance indicators), e.g., Precision-Sensitivity or Accuracy, one can seldom interpret the obtained numbers into the model's actual needs. We suggest generalizing Matthew's correlation coefficient into multi-dimensions. This generalization is based on a geometrical interpretation of the generalized confusion matrix.

Related papers

$F_β$-plot -- a visual tool for evaluating imbalanced data classifiers [0.0]
The paper proposes a simple approach to analyzing the popular parametric metric $F_beta$. It is possible to indicate for a given pool of analyzed classifiers when a given model should be preferred depending on user requirements.
arXiv Detail & Related papers (2024-04-11T18:07:57Z)
Correcting Flaws in Common Disentanglement Metrics [44.937838134027714]
In this paper, we identify two failings of existing metrics, which mean they can assign a high score to a model which is still entangled. We then consider the task of compositional generalization. Unlike prior works, we treat this as a classification problem, which allows us to use it to measure the disentanglement ability of the encoder. We show that performance on this task is (a) generally quite poor, (b) correlated with most disentanglement metrics, and (c) most strongly correlated with our newly proposed metrics.
arXiv Detail & Related papers (2023-04-05T09:43:58Z)
Rethinking Knowledge Graph Evaluation Under the Open-World Assumption [65.20527611711697]
Most knowledge graphs (KGs) are incomplete, which motivates one important research topic on automatically complementing knowledge graphs. Treating all unknown triplets as false is called the closed-world assumption. In this paper, we study KGC evaluation under a more realistic setting, namely the open-world assumption.
arXiv Detail & Related papers (2022-09-19T09:01:29Z)
Tuned Regularized Estimators for Linear Regression via Covariance Fitting [17.46329281993348]
We consider the problem of finding tuned regularized parameter estimators for linear models. We show that three known optimal linear estimators belong to a wider class of estimators. We show that the resulting class of estimators yields tuned versions of known regularized estimators.
arXiv Detail & Related papers (2022-01-21T16:08:08Z)
A moment-matching metric for latent variable generative models [0.0]
In scope of Goodhart's law, when a metric becomes a target it ceases to be a good metric. We propose a new metric for model comparison or regularization that relies on moments. It is common to draw samples from the fitted distribution when evaluating latent variable models.
arXiv Detail & Related papers (2021-10-04T17:51:08Z)
Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic View [82.80085730891126]
We provide the first modernally precise analysis of linear multiclass classification. Our analysis reveals that the classification accuracy is highly distribution-dependent. The insights gained may pave the way for a precise understanding of other classification algorithms.
arXiv Detail & Related papers (2020-11-16T05:17:29Z)
Benign overfitting in ridge regression [0.0]
We provide non-asymptotic generalization bounds for overparametrized ridge regression. We identify when small or negative regularization is sufficient for obtaining small generalization error.
arXiv Detail & Related papers (2020-09-29T20:00:31Z)
KPQA: A Metric for Generative Question Answering Using Keyphrase Weights [64.54593491919248]
KPQA-metric is a new metric for evaluating correctness of generative question answering systems. Our new metric assigns different weights to each token via keyphrase prediction. We show that our proposed metric has a significantly higher correlation with human judgments than existing metrics.
arXiv Detail & Related papers (2020-05-01T03:24:36Z)
Fractional norms and quasinorms do not help to overcome the curse of dimensionality [62.997667081978825]
Using of the Manhattan distance and even fractional quasinorms lp can help to overcome the curse of dimensionality in classification problems. A systematic comparison shows that the difference of the performance of kNN based on lp for p=2, 1, and 0.5 is statistically insignificant.
arXiv Detail & Related papers (2020-04-29T14:30:12Z)
Ambiguity in Sequential Data: Predicting Uncertain Futures with Recurrent Models [110.82452096672182]
We propose an extension of the Multiple Hypothesis Prediction (MHP) model to handle ambiguous predictions with sequential data. We also introduce a novel metric for ambiguous problems, which is better suited to account for uncertainties.
arXiv Detail & Related papers (2020-03-10T09:15:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.