Few-Shot Unsupervised Continual Learning through Meta-Examples
- URL: http://arxiv.org/abs/2009.08107v3
- Date: Tue, 17 Nov 2020 13:38:40 GMT
- Title: Few-Shot Unsupervised Continual Learning through Meta-Examples
- Authors: Alessia Bertugli, Stefano Vincenzi, Simone Calderara, Andrea Passerini
- Abstract summary: We introduce a novel and complex setting involving unsupervised meta-continual learning with unbalanced tasks.
We exploit a meta-learning scheme that simultaneously alleviates catastrophic forgetting and favors the generalization to new tasks.
Experimental results on few-shot learning benchmarks show competitive performance even compared to the supervised case.
- Score: 21.954394608030388
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In real-world applications, data do not reflect the ones commonly used for
neural networks training, since they are usually few, unlabeled and can be
available as a stream. Hence many existing deep learning solutions suffer from
a limited range of applications, in particular in the case of online streaming
data that evolve over time. To narrow this gap, in this work we introduce a
novel and complex setting involving unsupervised meta-continual learning with
unbalanced tasks. These tasks are built through a clustering procedure applied
to a fitted embedding space. We exploit a meta-learning scheme that
simultaneously alleviates catastrophic forgetting and favors the generalization
to new tasks. Moreover, to encourage feature reuse during the
meta-optimization, we exploit a single inner loop taking advantage of an
aggregated representation achieved through the use of a self-attention
mechanism. Experimental results on few-shot learning benchmarks show
competitive performance even compared to the supervised case. Additionally, we
empirically observe that in an unsupervised scenario, the small tasks and the
variability in the clusters pooling play a crucial role in the generalization
capability of the network. Further, on complex datasets, the exploitation of
more clusters than the true number of classes leads to higher results, even
compared to the ones obtained with full supervision, suggesting that a
predefined partitioning into classes can miss relevant structural information.
Related papers
- Disentangled Latent Spaces Facilitate Data-Driven Auxiliary Learning [15.41342100228504]
In deep learning, auxiliary objectives are often used to facilitate learning in situations where data is scarce.
We propose a novel framework, dubbed Detaux, whereby a weakly supervised disentanglement procedure is used to discover new unrelated classification tasks.
arXiv Detail & Related papers (2023-10-13T17:40:39Z) - A Distinct Unsupervised Reference Model From The Environment Helps
Continual Learning [5.332329421663282]
Open-Set Semi-Supervised Continual Learning (OSSCL) is a more realistic semi-supervised continual learning setting.
We present a model with two distinct parts: (i) the reference network captures general-purpose and task-agnostic knowledge in the environment by using a broad spectrum of unlabeled samples, and (ii) the learner network is designed to learn task-specific representations by exploiting supervised samples.
arXiv Detail & Related papers (2023-01-11T15:05:36Z) - A Multi-label Continual Learning Framework to Scale Deep Learning
Approaches for Packaging Equipment Monitoring [57.5099555438223]
We study multi-label classification in the continual scenario for the first time.
We propose an efficient approach that has a logarithmic complexity with regard to the number of tasks.
We validate our approach on a real-world multi-label Forecasting problem from the packaging industry.
arXiv Detail & Related papers (2022-08-08T15:58:39Z) - CHALLENGER: Training with Attribution Maps [63.736435657236505]
We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance.
In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.
arXiv Detail & Related papers (2022-05-30T13:34:46Z) - Learning Prototype-oriented Set Representations for Meta-Learning [85.19407183975802]
Learning from set-structured data is a fundamental problem that has recently attracted increasing attention.
This paper provides a novel optimal transport based way to improve existing summary networks.
We further instantiate it to the cases of few-shot classification and implicit meta generative modeling.
arXiv Detail & Related papers (2021-10-18T09:49:05Z) - Unsupervised Learning of Debiased Representations with Pseudo-Attributes [85.5691102676175]
We propose a simple but effective debiasing technique in an unsupervised manner.
We perform clustering on the feature embedding space and identify pseudoattributes by taking advantage of the clustering results.
We then employ a novel cluster-based reweighting scheme for learning debiased representation.
arXiv Detail & Related papers (2021-08-06T05:20:46Z) - Continual Learning From Unlabeled Data Via Deep Clustering [7.704949298975352]
Continual learning aims to learn new tasks incrementally using less computation and memory resources instead of retraining the model from scratch whenever new task arrives.
We introduce a new framework to make continual learning feasible in unsupervised mode by using pseudo label obtained from cluster assignments to update model.
arXiv Detail & Related papers (2021-04-14T23:46:17Z) - Continual Learning for Recurrent Neural Networks: a Review and Empirical
Evaluation [12.27992745065497]
Continual Learning with recurrent neural networks could pave the way to a large number of applications where incoming data is non stationary.
We organize the literature on CL for sequential data processing by providing a categorization of the contributions and a review of the benchmarks.
We propose two new benchmarks for CL with sequential data based on existing datasets, whose characteristics resemble real-world applications.
arXiv Detail & Related papers (2021-03-12T19:25:28Z) - ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for
Semi-supervised Continual Learning [52.831894583501395]
Continual learning assumes the incoming data are fully labeled, which might not be applicable in real applications.
We propose deep Online Replay with Discriminator Consistency (ORDisCo) to interdependently learn a classifier with a conditional generative adversarial network (GAN)
We show ORDisCo achieves significant performance improvement on various semi-supervised learning benchmark datasets for SSCL.
arXiv Detail & Related papers (2021-01-02T09:04:14Z) - Sense and Learn: Self-Supervision for Omnipresent Sensors [9.442811508809994]
We present a framework named Sense and Learn for representation or feature learning from raw sensory data.
It consists of several auxiliary tasks that can learn high-level and broadly useful features entirely from unannotated data without any human involvement in the tedious labeling process.
Our methodology achieves results that are competitive with the supervised approaches and close the gap through fine-tuning a network while learning the downstream tasks in most cases.
arXiv Detail & Related papers (2020-09-28T11:57:43Z) - CoLES: Contrastive Learning for Event Sequences with Self-Supervision [63.3568071938238]
We address the problem of self-supervised learning on discrete event sequences generated by real-world users.
We propose a new method "CoLES", which adapts contrastive learning, previously used for audio and computer vision domains.
arXiv Detail & Related papers (2020-02-19T15:15:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.