Evaluating the Predictive Performance of Positive-Unlabelled Classifiers: a brief critical review and practical recommendations for improvement

• URL: http://arxiv.org/abs/2206.02423v1
• Date: Mon, 6 Jun 2022 08:31:49 GMT
• Title: Evaluating the Predictive Performance of Positive-Unlabelled Classifiers: a brief critical review and practical recommendations for improvement
• Authors: Jack D. Saunders and Alex, A. Freitas
• Abstract summary: Positive-Unlabelled (PU) learning is a growing area of machine learning. This paper critically reviews the main PU learning evaluation approaches and the choice of predictive accuracy measures in 51 articles proposing PU classifiers.
• Score: 77.34726150561087
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Positive-Unlabelled (PU) learning is a growing area of machine learning that aims to learn classifiers from data consisting of labelled positive and unlabelled instances. Whilst much work has been done proposing methods for PU learning, little has been written on the subject of evaluating these methods. Many popular standard classification metrics cannot be precisely calculated due to the absence of fully labelled data, so alternative approaches must be taken. This short commentary paper critically reviews the main PU learning evaluation approaches and the choice of predictive accuracy measures in 51 articles proposing PU classifiers and provides practical recommendations for improvements in this area.

Related papers

• Probably Approximately Precision and Recall Learning [62.912015491907994]
  Precision and Recall are foundational metrics in machine learning. One-sided feedback--where only positive examples are observed during training--is inherent in many practical problems. We introduce a PAC learning framework where each hypothesis is represented by a graph, with edges indicating positive interactions.
  arXiv  Detail & Related papers  (2024-11-20T04:21:07Z)
• Meta-learning for Positive-unlabeled Classification [40.11462237689747]
  The proposed method minimizes the test classification risk after the model is adapted to PU data. The method embeds each instance into a task-specific space using neural networks. We empirically show that the proposed method outperforms existing methods with one synthetic and three real-world datasets.
  arXiv  Detail & Related papers  (2024-06-06T01:50:01Z)
• Soft Label PU Learning [9.66983903944539]
  We propose soft label PU learning, in which unlabeled data are assigned soft labels according to their probabilities of being positive. Considering the ground truth of TPR, FPR, and AUC, we then design PU counterparts of these metrics to evaluate the performances of soft label PU learning methods within validation data.
  arXiv  Detail & Related papers  (2024-05-03T10:46:19Z)
• Contrastive Approach to Prior Free Positive Unlabeled Learning [15.269090018352875]
  We propose a novel PU learning framework, that starts by learning a feature space through pretext-invariant representation learning. Our proposed approach handily outperforms state-of-the-art PU learning methods across several standard PU benchmark datasets.
  arXiv  Detail & Related papers  (2024-02-08T20:20:54Z)
• Evaluating Machine Unlearning via Epistemic Uncertainty [78.27542864367821]
  This work presents an evaluation of Machine Unlearning algorithms based on uncertainty. This is the first definition of a general evaluation of our best knowledge.
  arXiv  Detail & Related papers  (2022-08-23T09:37:31Z)
• Positive-Unlabeled Classification under Class-Prior Shift: A Prior-invariant Approach Based on Density Ratio Estimation [85.75352990739154]
  We propose a novel PU classification method based on density ratio estimation. A notable advantage of our proposed method is that it does not require the class-priors in the training phase.
  arXiv  Detail & Related papers  (2021-07-11T13:36:53Z)
• Learning the Truth From Only One Side of the Story [58.65439277460011]
  We focus on generalized linear models and show that without adjusting for this sampling bias, the model may converge suboptimally or even fail to converge to the optimal solution. We propose an adaptive approach that comes with theoretical guarantees and show that it outperforms several existing methods empirically.
  arXiv  Detail & Related papers  (2020-06-08T18:20:28Z)
• Active Bayesian Assessment for Black-Box Classifiers [20.668691047355072]
  We introduce an active Bayesian approach for assessment of classifier performance to satisfy the desiderata of both reliability and label-efficiency. We first develop inference strategies to quantify uncertainty for common assessment metrics such as accuracy, misclassification cost, and calibration error. We then propose a general framework for active Bayesian assessment using inferred uncertainty to guide efficient selection of instances for labeling.
  arXiv  Detail & Related papers  (2020-02-16T08:08:42Z)
• Efficient Policy Learning from Surrogate-Loss Classification Reductions [65.91730154730905]
  We consider the estimation problem given by a weighted surrogate-loss classification reduction of policy learning. We show that, under a correct specification assumption, the weighted classification formulation need not be efficient for policy parameters. We propose an estimation approach based on generalized method of moments, which is efficient for the policy parameters.
  arXiv  Detail & Related papers  (2020-02-12T18:54:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.