Leveraging Ensemble Diversity for Robust Self-Training in the Presence of Sample Selection Bias

• URL: http://arxiv.org/abs/2310.14814v4
• Date: Wed, 3 Apr 2024 09:32:09 GMT
• Title: Leveraging Ensemble Diversity for Robust Self-Training in the Presence of Sample Selection Bias
• Authors: Ambroise Odonnat, Vasilii Feofanov, Ievgen Redko,
• Abstract summary: Self-training is a well-known approach for semi-supervised learning. It consists of iteratively assigning pseudo-labels to unlabeled data for which the model is confident and treating them as labeled examples. For neural networks, softmax prediction probabilities are often used as a confidence measure, although they are known to be overconfident, even for wrong predictions. We propose a novel confidence measure, called $mathcalT$-similarity, built upon the prediction diversity of an ensemble of linear classifiers.
• Score: 5.698050337128548
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-training is a well-known approach for semi-supervised learning. It consists of iteratively assigning pseudo-labels to unlabeled data for which the model is confident and treating them as labeled examples. For neural networks, softmax prediction probabilities are often used as a confidence measure, although they are known to be overconfident, even for wrong predictions. This phenomenon is particularly intensified in the presence of sample selection bias, i.e., when data labeling is subject to some constraint. To address this issue, we propose a novel confidence measure, called $\mathcal{T}$-similarity, built upon the prediction diversity of an ensemble of linear classifiers. We provide the theoretical analysis of our approach by studying stationary points and describing the relationship between the diversity of the individual members and their performance. We empirically demonstrate the benefit of our confidence measure for three different pseudo-labeling policies on classification datasets of various data modalities. The code is available at https://github.com/ambroiseodt/tsim.

Related papers

• Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition [50.61991746981703]
  Current state-of-the-art LTSSL approaches rely on high-quality pseudo-labels for large-scale unlabeled data. This paper introduces a novel probabilistic framework that unifies various recent proposals in long-tail learning. We introduce a continuous contrastive learning method, CCL, extending our framework to unlabeled data using reliable and smoothed pseudo-labels.
  arXiv  Detail & Related papers  (2024-10-08T15:06:10Z)
• Pseudo Label Selection is a Decision Problem [0.0]
  Pseudo-Labeling is a simple and effective approach to semi-supervised learning. It requires criteria that guide the selection of pseudo-labeled data. Overfitting can be propagated to the final model by choosing instances with overconfident but wrong predictions.
  arXiv  Detail & Related papers  (2023-09-25T07:48:02Z)
• Probabilistic Test-Time Generalization by Variational Neighbor-Labeling [62.158807685159736]
  This paper strives for domain generalization, where models are trained exclusively on source domains before being deployed on unseen target domains. Probability pseudo-labeling of target samples to generalize the source-trained model to the target domain at test time. Variational neighbor labels that incorporate the information of neighboring target samples to generate more robust pseudo labels.
  arXiv  Detail & Related papers  (2023-07-08T18:58:08Z)
• Variational Classification [51.2541371924591]
  We derive a variational objective to train the model, analogous to the evidence lower bound (ELBO) used to train variational auto-encoders. Treating inputs to the softmax layer as samples of a latent variable, our abstracted perspective reveals a potential inconsistency. We induce a chosen latent distribution, instead of the implicit assumption found in a standard softmax layer.
  arXiv  Detail & Related papers  (2023-05-17T17:47:19Z)
• AggMatch: Aggregating Pseudo Labels for Semi-Supervised Learning [25.27527138880104]
  Semi-supervised learning has proven to be an effective paradigm for leveraging a huge amount of unlabeled data. We introduce AggMatch, which refines initial pseudo labels by using different confident instances. We conduct experiments to demonstrate the effectiveness of AggMatch over the latest methods on standard benchmarks.
  arXiv  Detail & Related papers  (2022-01-25T16:41:54Z)
• Multi-class Probabilistic Bounds for Self-learning [13.875239300089861]
  Pseudo-labeling is prone to error and runs the risk of adding noisy labels into unlabeled training data. We present a probabilistic framework for analyzing self-learning in the multi-class classification scenario with partially labeled data.
  arXiv  Detail & Related papers  (2021-09-29T13:57:37Z)
• Dash: Semi-Supervised Learning with Dynamic Thresholding [72.74339790209531]
  We propose a semi-supervised learning (SSL) approach that uses unlabeled examples to train models. Our proposed approach, Dash, enjoys its adaptivity in terms of unlabeled data selection.
  arXiv  Detail & Related papers  (2021-09-01T23:52:29Z)
• Minimax Active Learning [61.729667575374606]
  Active learning aims to develop label-efficient algorithms by querying the most representative samples to be labeled by a human annotator. Current active learning techniques either rely on model uncertainty to select the most uncertain samples or use clustering or reconstruction to choose the most diverse set of unlabeled examples. We develop a semi-supervised minimax entropy-based active learning algorithm that leverages both uncertainty and diversity in an adversarial manner.
  arXiv  Detail & Related papers  (2020-12-18T19:03:40Z)
• Robust Fairness under Covariate Shift [11.151913007808927]
  Making predictions that are fair with regard to protected group membership has become an important requirement for classification algorithms. We propose an approach that obtains the predictor that is robust to the worst-case in terms of target performance.
  arXiv  Detail & Related papers  (2020-10-11T04:42:01Z)
• How to trust unlabeled data? Instance Credibility Inference for Few-Shot Learning [47.21354101796544]
  This paper presents a statistical approach, dubbed Instance Credibility Inference (ICI) to exploit the support of unlabeled instances for few-shot visual recognition. We rank the credibility of pseudo-labeled instances along the regularization path of their corresponding incidental parameters, and the most trustworthy pseudo-labeled examples are preserved as the augmented labeled instances.
  arXiv  Detail & Related papers  (2020-07-15T03:38:09Z)
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
  Machine learning algorithms are susceptible to data poisoning attacks. We present a unifying view of randomized smoothing over arbitrary functions. We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
  arXiv  Detail & Related papers  (2020-02-07T21:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.