Self Training with Ensemble of Teacher Models

• URL: http://arxiv.org/abs/2107.08211v1
• Date: Sat, 17 Jul 2021 09:44:09 GMT
• Title: Self Training with Ensemble of Teacher Models
• Authors: Soumyadeep Ghosh, Sanjay Kumar, Janu Verma and Awanish Kumar
• Abstract summary: In order to train robust deep learning models, large amounts of labelled data is required. In the absence of such large repositories of labelled data, unlabeled data can be exploited for the same. Semi-Supervised learning aims to utilize such unlabeled data for training classification models.
• Score: 8.257085583227695
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In order to train robust deep learning models, large amounts of labelled data is required. However, in the absence of such large repositories of labelled data, unlabeled data can be exploited for the same. Semi-Supervised learning aims to utilize such unlabeled data for training classification models. Recent progress of self-training based approaches have shown promise in this area, which leads to this study where we utilize an ensemble approach for the same. A by-product of any semi-supervised approach may be loss of calibration of the trained model especially in scenarios where unlabeled data may contain out-of-distribution samples, which leads to this investigation on how to adapt to such effects. Our proposed algorithm carefully avoids common pitfalls in utilizing unlabeled data and leads to a more accurate and calibrated supervised model compared to vanilla self-training based student-teacher algorithms. We perform several experiments on the popular STL-10 database followed by an extensive analysis of our approach and study its effects on model accuracy and calibration.

Related papers

• Data Shapley in One Training Run [88.59484417202454]
  Data Shapley provides a principled framework for attributing data's contribution within machine learning contexts. Existing approaches require re-training models on different data subsets, which is computationally intensive. This paper introduces In-Run Data Shapley, which addresses these limitations by offering scalable data attribution for a target model of interest.
  arXiv  Detail & Related papers  (2024-06-16T17:09:24Z)
• Partially Blinded Unlearning: Class Unlearning for Deep Networks a Bayesian Perspective [4.31734012105466]
  Machine Unlearning is the process of selectively discarding information designated to specific sets or classes of data from a pre-trained model. We propose a methodology tailored for the purposeful elimination of information linked to a specific class of data from a pre-trained classification network. Our novel approach, termed textbfPartially-Blinded Unlearning (PBU), surpasses existing state-of-the-art class unlearning methods, demonstrating superior effectiveness.
  arXiv  Detail & Related papers  (2024-03-24T17:33:22Z)
• The Mirrored Influence Hypothesis: Efficient Data Influence Estimation by Harnessing Forward Passes [30.30769701138665]
  We introduce and explore the Mirrored Influence Hypothesis, highlighting a reciprocal nature of influence between training and test data. Specifically, it suggests that evaluating the influence of training data on test predictions can be reformulated as an equivalent, yet inverse problem. We introduce a new method for estimating the influence of training data, which requires calculating gradients for specific test samples, paired with a forward pass for each training point.
  arXiv  Detail & Related papers  (2024-02-14T03:43:05Z)
• Stubborn Lexical Bias in Data and Models [50.79738900885665]
  We use a new statistical method to examine whether spurious patterns in data appear in models trained on the data. We apply an optimization approach to *reweight* the training data, reducing thousands of spurious correlations. Surprisingly, though this method can successfully reduce lexical biases in the training data, we still find strong evidence of corresponding bias in the trained models.
  arXiv  Detail & Related papers  (2023-06-03T20:12:27Z)
• Certifying Data-Bias Robustness in Linear Regression [12.00314910031517]
  We present a technique for certifying whether linear regression models are pointwise-robust to label bias in a training dataset. We show how to solve this problem exactly for individual test points, and provide an approximate but more scalable method. We also unearth gaps in bias-robustness, such as high levels of non-robustness for certain bias assumptions on some datasets.
  arXiv  Detail & Related papers  (2022-06-07T20:47:07Z)
• Semi-supervised Deep Learning for Image Classification with Distribution Mismatch: A Survey [1.5469452301122175]
  Deep learning models rely on the abundance of labelled observations to train a prospective model. It is expensive to gather labelled observations of data, making the usage of deep learning models not ideal. In many situations different unlabelled data sources might be available. This raises the risk of a significant distribution mismatch between the labelled and unlabelled datasets.
  arXiv  Detail & Related papers  (2022-03-01T02:46:00Z)
• Harnessing Unlabeled Data to Improve Generalization of Biometric Gender and Age Classifiers [0.7874708385247353]
  Deep learning models need large amount of labeled data for model training and optimum parameter estimation. Due to privacy and security concerns, the large amount of labeled data could not be collected for certain applications such as those involving medical field. We propose self-ensemble based deep learning model that along with limited labeled data, harness unlabeled data for improving the generalization performance.
  arXiv  Detail & Related papers  (2021-10-09T01:06:01Z)
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning [78.83598532168256]
  Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
  arXiv  Detail & Related papers  (2021-04-11T09:50:24Z)
• Learning from others' mistakes: Avoiding dataset biases without modeling them [111.17078939377313]
  State-of-the-art natural language processing (NLP) models often learn to model dataset biases and surface form correlations instead of features that target the intended task. Previous work has demonstrated effective methods to circumvent these issues when knowledge of the bias is available. We show a method for training models that learn to ignore these problematic correlations.
  arXiv  Detail & Related papers  (2020-12-02T16:10:54Z)
• Uncertainty-aware Self-training for Text Classification with Few Labels [54.13279574908808]
  We study self-training as one of the earliest semi-supervised learning approaches to reduce the annotation bottleneck. We propose an approach to improve self-training by incorporating uncertainty estimates of the underlying neural network. We show our methods leveraging only 20-30 labeled samples per class for each task for training and for validation can perform within 3% of fully supervised pre-trained language models.
  arXiv  Detail & Related papers  (2020-06-27T08:13:58Z)
• How Training Data Impacts Performance in Learning-based Control [67.7875109298865]
  This paper derives an analytical relationship between the density of the training data and the control performance. We formulate a quality measure for the data set, which we refer to as $rho$-gap. We show how the $rho$-gap can be applied to a feedback linearizing control law.
  arXiv  Detail & Related papers  (2020-05-25T12:13:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.