Self-supervised visual feature learning with curriculum
- URL: http://arxiv.org/abs/2001.05634v1
- Date: Thu, 16 Jan 2020 03:28:58 GMT
- Title: Self-supervised visual feature learning with curriculum
- Authors: Vishal Keshav and Fabien Delattre
- Abstract summary: This paper takes inspiration from curriculum learning to progressively remove low level signals.
It shows that it significantly increase the speed of convergence of the downstream task.
- Score: 0.24366811507669126
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Self-supervised learning techniques have shown their abilities to learn
meaningful feature representation. This is made possible by training a model on
pretext tasks that only requires to find correlations between inputs or parts
of inputs. However, such pretext tasks need to be carefully hand selected to
avoid low level signals that could make those pretext tasks trivial. Moreover,
removing those shortcuts often leads to the loss of some semantically valuable
information. We show that it directly impacts the speed of learning of the
downstream task. In this paper we took inspiration from curriculum learning to
progressively remove low level signals and show that it significantly increase
the speed of convergence of the downstream task.
Related papers
- Ignorance is Bliss: Robust Control via Information Gating [60.17644038829572]
Informational parsimony provides a useful inductive bias for learning representations that achieve better generalization by being robust to noise and spurious correlations.
We propose textitinformation gating as a way to learn parsimonious representations that identify the minimal information required for a task.
arXiv Detail & Related papers (2023-03-10T18:31:50Z) - Task Compass: Scaling Multi-task Pre-training with Task Prefix [122.49242976184617]
Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks.
We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks.
Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
arXiv Detail & Related papers (2022-10-12T15:02:04Z) - Hierarchical Skills for Efficient Exploration [70.62309286348057]
In reinforcement learning, pre-trained low-level skills have the potential to greatly facilitate exploration.
Prior knowledge of the downstream task is required to strike the right balance between generality (fine-grained control) and specificity (faster learning) in skill design.
We propose a hierarchical skill learning framework that acquires skills of varying complexity in an unsupervised manner.
arXiv Detail & Related papers (2021-10-20T22:29:32Z) - Information-Preserving Contrastive Learning for Self-Supervised
Representations [42.74927142208983]
Contrastive learning is very effective at learning useful representations without supervision.
It may learn a shortcut that is irrelevant to the downstream task, and discard relevant information.
This paper presents information-preserving contrastive learning (IPCL), a new framework for unsupervised representation learning.
arXiv Detail & Related papers (2020-12-17T22:46:17Z) - Towards All-around Knowledge Transferring: Learning From Task-irrelevant
Labels [44.036667329736225]
Existing efforts mainly focus on transferring task-relevant knowledge from other similar data to tackle the issue.
To date, no large-scale studies have been performed to investigate the impact of task-irrelevant features.
We propose Task-Irrelevant Transfer Learning to exploit taskirrelevant features, which mainly are extracted from task-irrelevant labels.
arXiv Detail & Related papers (2020-11-17T06:43:58Z) - Usable Information and Evolution of Optimal Representations During
Training [79.38872675793813]
In particular, we find that semantically meaningful but ultimately irrelevant information is encoded in the early transient dynamics of training.
We show these effects on both perceptual decision-making tasks inspired by literature, as well as on standard image classification tasks.
arXiv Detail & Related papers (2020-10-06T03:50:19Z) - Predicting What You Already Know Helps: Provable Self-Supervised
Learning [60.27658820909876]
Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data.
We show a mechanism exploiting the statistical connections between certain em reconstruction-based pretext tasks that guarantee to learn a good representation.
We prove the linear layer yields small approximation error even for complex ground truth function class.
arXiv Detail & Related papers (2020-08-03T17:56:13Z) - Learn Faster and Forget Slower via Fast and Stable Task Adaptation [10.696114236900808]
Current fine-tuning techniques make pretrained models forget the transferred knowledge even before anything about the new task is learned.
We introduce Fast And Stable Task-adaptation (FAST), an easy to apply fine-tuning algorithm.
We empirically show that compared to prevailing fine-tuning practices, FAST learns the target task faster and forgets the source task slower.
arXiv Detail & Related papers (2020-07-02T21:13:55Z) - Learning Invariant Representations for Reinforcement Learning without
Reconstruction [98.33235415273562]
We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction.
Bisimulation metrics quantify behavioral similarity between states in continuous MDPs.
We demonstrate the effectiveness of our method at disregarding task-irrelevant information using modified visual MuJoCo tasks.
arXiv Detail & Related papers (2020-06-18T17:59:35Z) - Automatic Shortcut Removal for Self-Supervised Representation Learning [39.636691159890354]
In self-supervised visual representation learning, a feature extractor is trained on a "pretext task" for which labels can be generated cheaply, without human annotation.
Much work has gone into identifying such "shortcut" features and hand-designing schemes to reduce their effect.
We show that this assumption holds across common pretext tasks and datasets by training a "lens" network to make small image changes that maximally reduce performance in the pretext task.
arXiv Detail & Related papers (2020-02-20T16:00:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.