Realistic Evaluation of Deep Partial-Label Learning Algorithms

• URL: http://arxiv.org/abs/2502.10184v1
• Date: Fri, 14 Feb 2025 14:22:16 GMT
• Title: Realistic Evaluation of Deep Partial-Label Learning Algorithms
• Authors: Wei Wang, Dong-Dong Wu, Jindong Wang, Gang Niu, Min-Ling Zhang, Masashi Sugiyama,
• Abstract summary: Partial-label learning (PLL) is a weakly supervised learning problem in which each example is associated with multiple candidate labels and only one is the true label. In recent years, many deep algorithms have been developed to improve model performance. Some early developed algorithms are often underestimated and can outperform many later algorithms with complicated designs.
• Score: 94.79036193414058
• License:
• Abstract: Partial-label learning (PLL) is a weakly supervised learning problem in which each example is associated with multiple candidate labels and only one is the true label. In recent years, many deep PLL algorithms have been developed to improve model performance. However, we find that some early developed algorithms are often underestimated and can outperform many later algorithms with complicated designs. In this paper, we delve into the empirical perspective of PLL and identify several critical but previously overlooked issues. First, model selection for PLL is non-trivial, but has never been systematically studied. Second, the experimental settings are highly inconsistent, making it difficult to evaluate the effectiveness of the algorithms. Third, there is a lack of real-world image datasets that can be compatible with modern network architectures. Based on these findings, we propose PLENCH, the first Partial-Label learning bENCHmark to systematically compare state-of-the-art deep PLL algorithms. We investigate the model selection problem for PLL for the first time, and propose novel model selection criteria with theoretical guarantees. We also create Partial-Label CIFAR-10 (PLCIFAR10), an image dataset of human-annotated partial labels collected from Amazon Mechanical Turk, to provide a testbed for evaluating the performance of PLL algorithms in more realistic scenarios. Researchers can quickly and conveniently perform a comprehensive and fair evaluation and verify the effectiveness of newly developed algorithms based on PLENCH. We hope that PLENCH will facilitate standardized, fair, and practical evaluation of PLL algorithms in the future.

Related papers

• Probably Approximately Precision and Recall Learning [62.912015491907994]
  Precision and Recall are foundational metrics in machine learning. One-sided feedback--where only positive examples are observed during training--is inherent in many practical problems. We introduce a PAC learning framework where each hypothesis is represented by a graph, with edges indicating positive interactions.
  arXiv  Detail & Related papers  (2024-11-20T04:21:07Z)
• Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization [101.08992036691673]
  This paper explores a realistic unsupervised fine-tuning scenario, considering the presence of out-of-distribution samples from unknown classes. In particular, we focus on simultaneously enhancing out-of-distribution detection and the recognition of instances associated with known classes. We present a simple, efficient, and effective approach called Universal Entropy Optimization (UEO)
  arXiv  Detail & Related papers  (2023-08-24T16:47:17Z)
• CLImage: Human-Annotated Datasets for Complementary-Label Learning [8.335164415521838]
  We develop a protocol to collect complementary labels from human annotators. These datasets represent the very first real-world CLL datasets. We discover that the biased-nature of human-annotated complementary labels and the difficulty to validate with only complementary labels are outstanding barriers to practical CLL.
  arXiv  Detail & Related papers  (2023-05-15T01:48:53Z)
• ARNet: Automatic Refinement Network for Noisy Partial Label Learning [41.577081851679765]
  We propose a novel framework called "Automatic Refinement Network (ARNet)" Our method consists of multiple rounds. In each round, we purify the noisy samples through two key modules, i.e., noisy sample detection and label correction. We prove that our method is able to reduce the noise level of the dataset and eventually approximate the Bayes optimal.
  arXiv  Detail & Related papers  (2022-11-09T10:01:25Z)
• Few-Shot Partial-Label Learning [25.609766770479265]
  Partial-label learning (PLL) generally focuses on inducing a noise-tolerant multi-class by training on overly-annotated samples. Existing few-shot learning algorithms assume precise labels of the support set; as such, irrelevant labels may seriously mislead the meta-learner. In this paper, we introduce an approach called FsPLL (Few-shot image learning)
  arXiv  Detail & Related papers  (2021-06-02T07:03:54Z)
• Adaptive Sampling for Best Policy Identification in Markov Decision Processes [79.4957965474334]
  We investigate the problem of best-policy identification in discounted Markov Decision (MDPs) when the learner has access to a generative model. The advantages of state-of-the-art algorithms are discussed and illustrated.
  arXiv  Detail & Related papers  (2020-09-28T15:22:24Z)
• Provably Consistent Partial-Label Learning [120.4734093544867]
  Partial-label learning (PLL) is a multi-class classification problem, where each training example is associated with a set of candidate labels. In this paper, we propose the first generation model of candidate label sets, and develop two novel methods that are guaranteed to be consistent. Experiments on benchmark and real-world datasets validate the effectiveness of the proposed generation model and two methods.
  arXiv  Detail & Related papers  (2020-07-17T12:19:16Z)
• Progressive Identification of True Labels for Partial-Label Learning [112.94467491335611]
  Partial-label learning (PLL) is a typical weakly supervised learning problem, where each training instance is equipped with a set of candidate labels among which only one is the true label. Most existing methods elaborately designed as constrained optimizations that must be solved in specific manners, making their computational complexity a bottleneck for scaling up to big data. This paper proposes a novel framework of classifier with flexibility on the model and optimization algorithm.
  arXiv  Detail & Related papers  (2020-02-19T08:35:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.