Policy-Induced Self-Supervision Improves Representation Finetuning in
Visual RL
- URL: http://arxiv.org/abs/2302.06009v1
- Date: Sun, 12 Feb 2023 21:52:28 GMT
- Title: Policy-Induced Self-Supervision Improves Representation Finetuning in
Visual RL
- Authors: S\'ebastien M. R. Arnold, Fei Sha
- Abstract summary: We study how to transfer representations pretrained on source tasks to target tasks in visual percept based RL.
We analyze two popular approaches: freezing or finetuning the pretrained representations.
- Score: 19.32387263597031
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We study how to transfer representations pretrained on source tasks to target
tasks in visual percept based RL. We analyze two popular approaches: freezing
or finetuning the pretrained representations. Empirical studies on a set of
popular tasks reveal several properties of pretrained representations. First,
finetuning is required even when pretrained representations perfectly capture
the information required to solve the target task. Second, finetuned
representations improve learnability and are more robust to noise. Third,
pretrained bottom layers are task-agnostic and readily transferable to new
tasks, while top layers encode task-specific information and require
adaptation. Building on these insights, we propose a self-supervised objective
that clusters representations according to the policy they induce, as opposed
to traditional representation similarity measures which are policy-agnostic
(e.g. Euclidean norm, cosine similarity). Together with freezing the bottom
layers, this objective results in significantly better representation than
frozen, finetuned, and self-supervised alternatives on a wide range of
benchmarks.
Related papers
- ACTRESS: Active Retraining for Semi-supervised Visual Grounding [52.08834188447851]
A previous study, RefTeacher, makes the first attempt to tackle this task by adopting the teacher-student framework to provide pseudo confidence supervision and attention-based supervision.
This approach is incompatible with current state-of-the-art visual grounding models, which follow the Transformer-based pipeline.
Our paper proposes the ACTive REtraining approach for Semi-Supervised Visual Grounding, abbreviated as ACTRESS.
arXiv Detail & Related papers (2024-07-03T16:33:31Z) - Intra-task Mutual Attention based Vision Transformer for Few-Shot Learning [12.5354658533836]
Humans possess remarkable ability to accurately classify new, unseen images after being exposed to only a few examples.
For artificial neural network models, determining the most relevant features for distinguishing between two images with limited samples presents a challenge.
We propose an intra-task mutual attention method for few-shot learning, that involves splitting the support and query samples into patches.
arXiv Detail & Related papers (2024-05-06T02:02:57Z) - Foundation Policies with Hilbert Representations [54.44869979017766]
We propose an unsupervised framework to pre-train generalist policies from unlabeled offline data.
Our key insight is to learn a structured representation that preserves the temporal structure of the underlying environment.
Our experiments show that our unsupervised policies can solve goal-conditioned and general RL tasks in a zero-shot fashion.
arXiv Detail & Related papers (2024-02-23T19:09:10Z) - Representation Abstractions as Incentives for Reinforcement Learning
Agents: A Robotic Grasping Case Study [3.4777703321218225]
This work examines the effect of various state representations in incentivizing the agent to solve a specific robotic task.
A continuum of state representation abstractions is defined, starting from a model-based approach with complete system knowledge.
We examine the effects of each representation in the ability of the agent to solve the task in simulation and the transferability of the learned policy to the real robot.
arXiv Detail & Related papers (2023-09-21T11:41:22Z) - Patch-Prompt Aligned Bayesian Prompt Tuning for Vision-Language Models [48.77653835765705]
We introduce a probabilistic resolution to prompt tuning, where the label-specific prompts are generated hierarchically by first sampling a latent vector from an underlying distribution and then employing a lightweight generative model.
We evaluate the effectiveness of our approach on four tasks: few-shot image recognition, base-to-new generalization, dataset transfer learning, and domain shifts.
arXiv Detail & Related papers (2023-03-16T06:09:15Z) - Self-Supervised Learning via Maximum Entropy Coding [57.56570417545023]
We propose Maximum Entropy Coding (MEC) as a principled objective that explicitly optimize on the structure of the representation.
MEC learns a more generalizable representation than previous methods based on specific pretext tasks.
It achieves state-of-the-art performance consistently on various downstream tasks, including not only ImageNet linear probe, but also semi-supervised classification, object detection, instance segmentation, and object tracking.
arXiv Detail & Related papers (2022-10-20T17:58:30Z) - Explaining, Evaluating and Enhancing Neural Networks' Learned
Representations [2.1485350418225244]
We show how explainability can be an aid, rather than an obstacle, towards better and more efficient representations.
We employ such attributions to define two novel scores for evaluating the informativeness and the disentanglement of latent embeddings.
We show that adopting our proposed scores as constraints during the training of a representation learning task improves the downstream performance of the model.
arXiv Detail & Related papers (2022-02-18T19:00:01Z) - Learning to Relate Depth and Semantics for Unsupervised Domain
Adaptation [87.1188556802942]
We present an approach for encoding visual task relationships to improve model performance in an Unsupervised Domain Adaptation (UDA) setting.
We propose a novel Cross-Task Relation Layer (CTRL), which encodes task dependencies between the semantic and depth predictions.
Furthermore, we propose an Iterative Self-Learning (ISL) training scheme, which exploits semantic pseudo-labels to provide extra supervision on the target domain.
arXiv Detail & Related papers (2021-05-17T13:42:09Z) - Conditional Meta-Learning of Linear Representations [57.90025697492041]
Standard meta-learning for representation learning aims to find a common representation to be shared across multiple tasks.
In this work we overcome this issue by inferring a conditioning function, mapping the tasks' side information into a representation tailored to the task at hand.
We propose a meta-algorithm capable of leveraging this advantage in practice.
arXiv Detail & Related papers (2021-03-30T12:02:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.