Uncertainty-aware Sampling for Long-tailed Semi-supervised Learning

• URL: http://arxiv.org/abs/2401.04435v1
• Date: Tue, 9 Jan 2024 08:59:39 GMT
• Title: Uncertainty-aware Sampling for Long-tailed Semi-supervised Learning
• Authors: Kuo Yang, Duo Li, Menghan Hu, Guangtao Zhai, Xiaokang Yang, Xiao-Ping Zhang
• Abstract summary: We introduce uncertainty into the modeling process for pseudo-label sampling, taking into account that the model performance on the tailed classes varies over different training stages. This approach allows the model to perceive the uncertainty of pseudo-labels at different training stages, thereby adaptively adjusting the selection thresholds for different classes. Compared to other methods such as the baseline method FixMatch, UDTS achieves an increase in accuracy of at least approximately 5.26%, 1.75%, 9.96%, and 1.28% on the natural scene image datasets.
• Score: 89.98353600316285
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For semi-supervised learning with imbalance classes, the long-tailed distribution of data will increase the model prediction bias toward dominant classes, undermining performance on less frequent classes. Existing methods also face challenges in ensuring the selection of sufficiently reliable pseudo-labels for model training and there is a lack of mechanisms to adjust the selection of more reliable pseudo-labels based on different training stages. To mitigate this issue, we introduce uncertainty into the modeling process for pseudo-label sampling, taking into account that the model performance on the tailed classes varies over different training stages. For example, at the early stage of model training, the limited predictive accuracy of model results in a higher rate of uncertain pseudo-labels. To counter this, we propose an Uncertainty-Aware Dynamic Threshold Selection (UDTS) approach. This approach allows the model to perceive the uncertainty of pseudo-labels at different training stages, thereby adaptively adjusting the selection thresholds for different classes. Compared to other methods such as the baseline method FixMatch, UDTS achieves an increase in accuracy of at least approximately 5.26%, 1.75%, 9.96%, and 1.28% on the natural scene image datasets CIFAR10-LT, CIFAR100-LT, STL-10-LT, and the medical image dataset TissueMNIST, respectively. The source code of UDTS is publicly available at: https://github.com/yangk/UDTS.

Related papers

• Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognition [50.61991746981703]
  Current state-of-the-art LTSSL approaches rely on high-quality pseudo-labels for large-scale unlabeled data. This paper introduces a novel probabilistic framework that unifies various recent proposals in long-tail learning. We introduce a continuous contrastive learning method, CCL, extending our framework to unlabeled data using reliable and smoothed pseudo-labels.
  arXiv  Detail & Related papers  (2024-10-08T15:06:10Z)
• Twice Class Bias Correction for Imbalanced Semi-Supervised Learning [59.90429949214134]
  We introduce a novel approach called textbfTwice textbfClass textbfBias textbfCorrection (textbfTCBC) We estimate the class bias of the model parameters during the training process. We apply a secondary correction to the model's pseudo-labels for unlabeled samples.
  arXiv  Detail & Related papers  (2023-12-27T15:06:36Z)
• Geometry-Aware Adaptation for Pretrained Models [15.715395029966812]
  We propose a drop-in replacement of the standard prediction rule, swapping argmax with the Fr'echet mean. Our proposed approach, Loki, gains up to 29.7% relative improvement over SimCLR on ImageNet. When no such metric is available, Loki can use self-derived metrics from class embeddings and obtains a 10.5% improvement on pretrained zero-shot models.
  arXiv  Detail & Related papers  (2023-07-23T04:48:41Z)
• Post-hoc Uncertainty Learning using a Dirichlet Meta-Model [28.522673618527417]
  We propose a novel Bayesian meta-model to augment pre-trained models with better uncertainty quantification abilities. Our proposed method requires no additional training data and is flexible enough to quantify different uncertainties. We demonstrate our proposed meta-model approach's flexibility and superior empirical performance on these applications.
  arXiv  Detail & Related papers  (2022-12-14T17:34:11Z)
• Selective Classification Via Neural Network Training Dynamics [26.58209894993386]
  We show that state-of-the-art selective classification performance can be attained solely from studying the training dynamics of a model. Our method achieves state-of-the-art accuracy/coverage trade-offs on typical selective classification benchmarks.
  arXiv  Detail & Related papers  (2022-05-26T17:51:29Z)
• Improving Calibration for Long-Tailed Recognition [68.32848696795519]
  We propose two methods to improve calibration and performance in such scenarios. For dataset bias due to different samplers, we propose shifted batch normalization. Our proposed methods set new records on multiple popular long-tailed recognition benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-01T13:55:21Z)
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning [53.1047775185362]
  Pseudo-labeling (PL) is a general SSL approach that does not have this constraint but performs relatively poorly in its original formulation. We argue that PL underperforms due to the erroneous high confidence predictions from poorly calibrated models. We propose an uncertainty-aware pseudo-label selection (UPS) framework which improves pseudo labeling accuracy by drastically reducing the amount of noise encountered in the training process.
  arXiv  Detail & Related papers  (2021-01-15T23:29:57Z)
• Uncertainty-aware Self-training for Text Classification with Few Labels [54.13279574908808]
  We study self-training as one of the earliest semi-supervised learning approaches to reduce the annotation bottleneck. We propose an approach to improve self-training by incorporating uncertainty estimates of the underlying neural network. We show our methods leveraging only 20-30 labeled samples per class for each task for training and for validation can perform within 3% of fully supervised pre-trained language models.
  arXiv  Detail & Related papers  (2020-06-27T08:13:58Z)
• Mitigating Class Boundary Label Uncertainty to Reduce Both Model Bias and Variance [4.563176550691304]
  We investigate a new approach to handle inaccuracy and uncertainty in the training data labels. Our method can reduce both bias and variance by estimating the pointwise label uncertainty of the training set.
  arXiv  Detail & Related papers  (2020-02-23T18:24:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.