Self-Supervised Learning from Semantically Imprecise Data
- URL: http://arxiv.org/abs/2104.10901v1
- Date: Thu, 22 Apr 2021 07:26:14 GMT
- Title: Self-Supervised Learning from Semantically Imprecise Data
- Authors: Clemens-Alexander Brust, Bj\"orn Barz, Joachim Denzler
- Abstract summary: Learning from imprecise labels such as "animal" or "bird" is an important capability when expertly labeled training data is scarce.
CHILLAX is a recently proposed method to tackle this task.
We extend CHILLAX with a self-supervised scheme using constrained extrapolation to generate pseudo-labels.
- Score: 7.24935792316121
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Learning from imprecise labels such as "animal" or "bird", but making precise
predictions like "snow bunting" at test time is an important capability when
expertly labeled training data is scarce. Contributions by volunteers or
results of web crawling lack precision in this manner, but are still valuable.
And crucially, these weakly labeled examples are available in larger quantities
for lower cost than high-quality bespoke training data.
CHILLAX, a recently proposed method to tackle this task, leverages a
hierarchical classifier to learn from imprecise labels. However, it has two
major limitations. First, it is not capable of learning from effectively
unlabeled examples at the root of the hierarchy, e.g. "object". Second, an
extrapolation of annotations to precise labels is only performed at test time,
where confident extrapolations could be already used as training data.
In this work, we extend CHILLAX with a self-supervised scheme using
constrained extrapolation to generate pseudo-labels. This addresses the second
concern, which in turn solves the first problem, enabling an even weaker
supervision requirement than CHILLAX. We evaluate our approach empirically and
show that our method allows for a consistent accuracy improvement of 0.84 to
1.19 percent points over CHILLAX and is suitable as a drop-in replacement
without any negative consequences such as longer training times.
Related papers
- Comprehensive Benchmarking of Entropy and Margin Based Scoring Metrics
for Data Selection [0.8192907805418583]
We present ways of assessing prospective training examples for their "usefulness" or "difficulty"
We primarily experiment with entropy and Error L2-Norm (EL2N) scores.
We then conduct training data augmentation experiments using these de-identified datasets and demonstrate that score-based selection can result in a 2% decrease in semantic error rate and 4%-7% decrease in domain classification error rate when compared to the baseline technique of random selection.
arXiv Detail & Related papers (2023-11-27T20:33:54Z) - Boosting Semi-Supervised Learning by bridging high and low-confidence
predictions [4.18804572788063]
Pseudo-labeling is a crucial technique in semi-supervised learning (SSL)
We propose a new method called ReFixMatch, which aims to utilize all of the unlabeled data during training.
arXiv Detail & Related papers (2023-08-15T00:27:18Z) - Shrinking Class Space for Enhanced Certainty in Semi-Supervised Learning [59.44422468242455]
We propose a novel method dubbed ShrinkMatch to learn uncertain samples.
For each uncertain sample, it adaptively seeks a shrunk class space, which merely contains the original top-1 class.
We then impose a consistency regularization between a pair of strongly and weakly augmented samples in the shrunk space to strive for discriminative representations.
arXiv Detail & Related papers (2023-08-13T14:05:24Z) - Are labels informative in semi-supervised learning? -- Estimating and
leveraging the missing-data mechanism [4.675583319625962]
Semi-supervised learning is a powerful technique for leveraging unlabeled data to improve machine learning models.
It can be affected by the presence of informative'' labels, which occur when some classes are more likely to be labeled than others.
We propose a novel approach to address this issue by estimating the missing-data mechanism and using inverse propensity weighting to debias any SSL algorithm.
arXiv Detail & Related papers (2023-02-15T09:18:46Z) - Weighted Distillation with Unlabeled Examples [15.825078347452024]
Distillation with unlabeled examples is a popular and powerful method for training deep neural networks in settings where the amount of labeled data is limited.
This paper proposes a principled approach for addressing this issue based on a ''debiasing'' reweighting of the student's loss function tailored to the distillation training paradigm.
arXiv Detail & Related papers (2022-10-13T04:08:56Z) - Debiased Pseudo Labeling in Self-Training [77.83549261035277]
Deep neural networks achieve remarkable performances on a wide range of tasks with the aid of large-scale labeled datasets.
To mitigate the requirement for labeled data, self-training is widely used in both academia and industry by pseudo labeling on readily-available unlabeled data.
We propose Debiased, in which the generation and utilization of pseudo labels are decoupled by two independent heads.
arXiv Detail & Related papers (2022-02-15T02:14:33Z) - Dash: Semi-Supervised Learning with Dynamic Thresholding [72.74339790209531]
We propose a semi-supervised learning (SSL) approach that uses unlabeled examples to train models.
Our proposed approach, Dash, enjoys its adaptivity in terms of unlabeled data selection.
arXiv Detail & Related papers (2021-09-01T23:52:29Z) - A Theoretical Analysis of Learning with Noisily Labeled Data [62.946840431501855]
We first show that in the first epoch training, the examples with clean labels will be learned first.
We then show that after the learning from clean data stage, continuously training model can achieve further improvement in testing error.
arXiv Detail & Related papers (2021-04-08T23:40:02Z) - Are Fewer Labels Possible for Few-shot Learning? [81.89996465197392]
Few-shot learning is challenging due to its very limited data and labels.
Recent studies in big transfer (BiT) show that few-shot learning can greatly benefit from pretraining on large scale labeled dataset in a different domain.
We propose eigen-finetuning to enable fewer shot learning by leveraging the co-evolution of clustering and eigen-samples in the finetuning.
arXiv Detail & Related papers (2020-12-10T18:59:29Z) - Don't Wait, Just Weight: Improving Unsupervised Representations by
Learning Goal-Driven Instance Weights [92.16372657233394]
Self-supervised learning techniques can boost performance by learning useful representations from unlabelled data.
We show that by learning Bayesian instance weights for the unlabelled data, we can improve the downstream classification accuracy.
Our method, BetaDataWeighter is evaluated using the popular self-supervised rotation prediction task on STL-10 and Visual Decathlon.
arXiv Detail & Related papers (2020-06-22T15:59:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.