FROST: Faster and more Robust One-shot Semi-supervised Training
- URL: http://arxiv.org/abs/2011.09471v4
- Date: Fri, 4 Dec 2020 14:04:18 GMT
- Title: FROST: Faster and more Robust One-shot Semi-supervised Training
- Authors: Helena E. Liu and Leslie N. Smith
- Abstract summary: We present a one-shot semi-supervised learning method that trains up to an order of magnitude faster and is more robust than state-of-the-art methods.
Our experiments demonstrate FROST's capability to perform well when the composition of the unlabeled data is unknown.
- Score: 0.0
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Recent advances in one-shot semi-supervised learning have lowered the barrier
for deep learning of new applications. However, the state-of-the-art for
semi-supervised learning is slow to train and the performance is sensitive to
the choices of the labeled data and hyper-parameter values. In this paper, we
present a one-shot semi-supervised learning method that trains up to an order
of magnitude faster and is more robust than state-of-the-art methods.
Specifically, we show that by combining semi-supervised learning with a
one-stage, single network version of self-training, our FROST methodology
trains faster and is more robust to choices for the labeled samples and changes
in hyper-parameters. Our experiments demonstrate FROST's capability to perform
well when the composition of the unlabeled data is unknown; that is when the
unlabeled data contain unequal numbers of each class and can contain
out-of-distribution examples that don't belong to any of the training classes.
High performance, speed of training, and insensitivity to hyper-parameters make
FROST the most practical method for one-shot semi-supervised training. Our code
is available at https://github.com/HelenaELiu/FROST.
Related papers
- One-bit Supervision for Image Classification: Problem, Solution, and
Beyond [114.95815360508395]
This paper presents one-bit supervision, a novel setting of learning with fewer labels, for image classification.
We propose a multi-stage training paradigm and incorporate negative label suppression into an off-the-shelf semi-supervised learning algorithm.
In multiple benchmarks, the learning efficiency of the proposed approach surpasses that using full-bit, semi-supervised supervision.
arXiv Detail & Related papers (2023-11-26T07:39:00Z) - Fast Propagation is Better: Accelerating Single-Step Adversarial
Training via Sampling Subnetworks [69.54774045493227]
A drawback of adversarial training is the computational overhead introduced by the generation of adversarial examples.
We propose to exploit the interior building blocks of the model to improve efficiency.
Compared with previous methods, our method not only reduces the training cost but also achieves better model robustness.
arXiv Detail & Related papers (2023-10-24T01:36:20Z) - SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for
Accelerating BERT Inference [18.456002674399244]
We propose a novel dynamic early exiting combined with layer skipping for BERT inference named SmartBERT.
SmartBERT can adaptively skip some layers and adaptively choose whether to exit.
We conduct experiments on eight classification datasets of the GLUE benchmark.
arXiv Detail & Related papers (2023-03-16T12:44:16Z) - Boosting Low-Data Instance Segmentation by Unsupervised Pre-training
with Saliency Prompt [103.58323875748427]
This work offers a novel unsupervised pre-training solution for low-data regimes.
Inspired by the recent success of the Prompting technique, we introduce a new pre-training method that boosts QEIS models.
Experimental results show that our method significantly boosts several QEIS models on three datasets.
arXiv Detail & Related papers (2023-02-02T15:49:03Z) - Distributed Adversarial Training to Robustify Deep Neural Networks at
Scale [100.19539096465101]
Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification.
To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training.
We propose a large-batch adversarial training framework implemented over multiple machines.
arXiv Detail & Related papers (2022-06-13T15:39:43Z) - MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the
Edge [72.16021611888165]
This paper proposes a novel Memory-Economic Sparse Training (MEST) framework targeting for accurate and fast execution on edge devices.
The proposed MEST framework consists of enhancements by Elastic Mutation (EM) and Soft Memory Bound (&S)
Our results suggest that unforgettable examples can be identified in-situ even during the dynamic exploration of sparsity masks.
arXiv Detail & Related papers (2021-10-26T21:15:17Z) - Self-Progressing Robust Training [146.8337017922058]
Current robust training methods such as adversarial training explicitly uses an "attack" to generate adversarial examples.
We propose a new framework called SPROUT, self-progressing robust training.
Our results shed new light on scalable, effective and attack-independent robust training methods.
arXiv Detail & Related papers (2020-12-22T00:45:24Z) - MUSCLE: Strengthening Semi-Supervised Learning Via Concurrent
Unsupervised Learning Using Mutual Information Maximization [29.368950377171995]
We introduce Mutual-information-based Unsupervised & Semi-supervised Concurrent LEarning (MUSCLE) to combine both unsupervised and semi-supervised learning.
MUSCLE can be used as a stand-alone training scheme for neural networks, and can also be incorporated into other learning approaches.
We show that the proposed hybrid model outperforms state of the art on several standard benchmarks, including CIFAR-10, CIFAR-100, and Mini-Imagenet.
arXiv Detail & Related papers (2020-11-30T23:01:04Z) - Building One-Shot Semi-supervised (BOSS) Learning up to Fully Supervised
Performance [0.0]
We show the potential for building one-shot semi-supervised (BOSS) learning on Cifar-10 and SVHN.
Our method combines class prototype refining, class balancing, and self-training.
Rigorous empirical evaluations provide evidence that labeling large datasets is not necessary for training deep neural networks.
arXiv Detail & Related papers (2020-06-16T17:56:00Z) - Empirical Perspectives on One-Shot Semi-supervised Learning [0.0]
One of the greatest obstacles in the adoption of deep neural networks for new applications is that training the network typically requires a large number of manually labeled training samples.
We empirically investigate the scenario where one has access to large amounts of unlabeled data but require labeling only a single sample per class in order to train a deep network.
arXiv Detail & Related papers (2020-04-08T17:51:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.