Vision-Language Models Provide Promptable Representations for Reinforcement Learning

• URL: http://arxiv.org/abs/2402.02651v3
• Date: Thu, 23 May 2024 01:04:11 GMT
• Title: Vision-Language Models Provide Promptable Representations for Reinforcement Learning
• Authors: William Chen, Oier Mees, Aviral Kumar, Sergey Levine,
• Abstract summary: We propose a novel approach that uses the vast amounts of general and indexable world knowledge encoded in vision-language models (VLMs) pre-trained on Internet-scale data for embodied reinforcement learning (RL) We show that our approach can use chain-of-thought prompting to produce representations of common-sense semantic reasoning, improving policy performance in novel scenes by 1.5 times.
• Score: 67.40524195671479
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Humans can quickly learn new behaviors by leveraging background world knowledge. In contrast, agents trained with reinforcement learning (RL) typically learn behaviors from scratch. We thus propose a novel approach that uses the vast amounts of general and indexable world knowledge encoded in vision-language models (VLMs) pre-trained on Internet-scale data for embodied RL. We initialize policies with VLMs by using them as promptable representations: embeddings that encode semantic features of visual observations based on the VLM's internal knowledge and reasoning capabilities, as elicited through prompts that provide task context and auxiliary information. We evaluate our approach on visually-complex, long horizon RL tasks in Minecraft and robot navigation in Habitat. We find that our policies trained on embeddings from off-the-shelf, general-purpose VLMs outperform equivalent policies trained on generic, non-promptable image embeddings. We also find our approach outperforms instruction-following methods and performs comparably to domain-specific embeddings. Finally, we show that our approach can use chain-of-thought prompting to produce representations of common-sense semantic reasoning, improving policy performance in novel scenes by 1.5 times.

Related papers

• Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models [33.504700578933424]
  Low sample efficiency is an enduring challenge of reinforcement learning (RL) We introduce a framework that harnesses large language models to extract background knowledge of an environment. Our experiments show that these methods achieve significant sample efficiency improvements in a spectrum of downstream tasks.
  arXiv  Detail & Related papers  (2024-07-04T14:33:47Z)
• Optimization of Prompt Learning via Multi-Knowledge Representation for Vision-Language Models [26.964848679914354]
  CoKnow is a framework that enhances Prompt Learning for Vision-Language Models with rich contextual knowledge. We conducted extensive experiments on 11 publicly available datasets, demonstrating that CoKnow outperforms a series of previous methods.
  arXiv  Detail & Related papers  (2024-04-16T07:44:52Z)
• How Can LLM Guide RL? A Value-Based Approach [68.55316627400683]
  Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. Recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities. We develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning.
  arXiv  Detail & Related papers  (2024-02-25T20:07:13Z)
• Context-Aware Prompt Tuning for Vision-Language Model with Dual-Alignment [15.180715595425864]
  We introduce a novel method to improve the prompt learning of vision-language models by incorporating pre-trained large language models (LLMs) With DuAl-PT, we propose to learn more context-aware prompts, benefiting from both explicit and implicit context modeling. Empirically, DuAl-PT achieves superior performance on 11 downstream datasets on few-shot recognition and base-to-new generalization.
  arXiv  Detail & Related papers  (2023-09-08T06:51:15Z)
• SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks [52.766795949716986]
  We present a study of the generalization capabilities of the pre-trained visual representations at the categorical level. We propose SpawnNet, a novel two-stream architecture that learns to fuse pre-trained multi-layer representations into a separate network to learn a robust policy.
  arXiv  Detail & Related papers  (2023-07-07T13:01:29Z)
• ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
  We introduce Action-Aware Embodied Learning for Perception (ALP) ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. We show that ALP outperforms existing baselines in several downstream perception tasks.
  arXiv  Detail & Related papers  (2023-06-16T21:51:04Z)
• Jump-Start Reinforcement Learning [68.82380421479675]
  We present a meta algorithm that can use offline data, demonstrations, or a pre-existing policy to initialize an RL policy. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks. We show via experiments that JSRL is able to significantly outperform existing imitation and reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2022-04-05T17:25:22Z)
• Embodied Learning for Lifelong Visual Perception [33.02424587900808]
  We study lifelong visual perception in an embodied setup, where we develop new models and compare various agents that navigate in buildings. The purpose of the agents is to recognize objects and other semantic classes in the whole building at the end of a process that combines exploration and active visual learning.
  arXiv  Detail & Related papers  (2021-12-28T10:47:13Z)
• Robust Deep Reinforcement Learning via Multi-View Information Bottleneck [7.188571996124112]
  We introduce an auxiliary objective based on the multi-view information bottleneck (MIB) principle. This encourages learning representations that are both predictive of the future and less sensitive to task-irrelevant distractions. We demonstrate that our approach can achieve SOTA performance on challenging visual control tasks, even when the background is replaced with natural videos.
  arXiv  Detail & Related papers  (2021-02-26T02:24:36Z)
• Teaching with Commentaries [108.62722733649542]
  We propose a flexible teaching framework using commentaries and learned meta-information. We find that commentaries can improve training speed and/or performance. commentaries can be reused when training new models to obtain performance benefits.
  arXiv  Detail & Related papers  (2020-11-05T18:52:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.