Generalization to New Sequential Decision Making Tasks with In-Context Learning

• URL: http://arxiv.org/abs/2312.03801v1
• Date: Wed, 6 Dec 2023 15:19:28 GMT
• Title: Generalization to New Sequential Decision Making Tasks with In-Context Learning
• Authors: Sharath Chandra Raparthy, Eric Hambro, Robert Kirk, Mikael Henaff, Roberta Raileanu
• Abstract summary: Training autonomous agents that can learn new tasks from only a handful of demonstrations is a long-standing problem in machine learning. In this paper, we show that naively applying transformers to sequential decision making problems does not enable in-context learning of new tasks. We investigate different design choices and find that larger model and dataset sizes, as well as more task diversity, environmentity, and trajectory burstiness, all result in better in-context learning of new out-of-distribution tasks.
• Score: 23.36106067650874
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training autonomous agents that can learn new tasks from only a handful of demonstrations is a long-standing problem in machine learning. Recently, transformers have been shown to learn new language or vision tasks without any weight updates from only a few examples, also referred to as in-context learning. However, the sequential decision making setting poses additional challenges having a lower tolerance for errors since the environment's stochasticity or the agent's actions can lead to unseen, and sometimes unrecoverable, states. In this paper, we use an illustrative example to show that naively applying transformers to sequential decision making problems does not enable in-context learning of new tasks. We then demonstrate how training on sequences of trajectories with certain distributional properties leads to in-context learning of new sequential decision making tasks. We investigate different design choices and find that larger model and dataset sizes, as well as more task diversity, environment stochasticity, and trajectory burstiness, all result in better in-context learning of new out-of-distribution tasks. By training on large diverse offline datasets, our model is able to learn new MiniHack and Procgen tasks without any weight updates from just a handful of demonstrations.

Related papers

• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Look-Ahead Selective Plasticity for Continual Learning of Visual Tasks [9.82510084910641]
  We propose a new mechanism that takes place during task boundaries, i.e., when one task finishes and another starts. We evaluate the proposed methods on benchmark computer vision datasets including CIFAR10 and TinyImagenet.
  arXiv  Detail & Related papers  (2023-11-02T22:00:23Z)
• Few-Shot In-Context Imitation Learning via Implicit Graph Alignment [15.215659641228655]
  We formulate imitation learning as a conditional alignment problem between graph representations of objects. We show that this conditioning allows for in-context learning, where a robot can perform a task on a set of new objects immediately after the demonstrations.
  arXiv  Detail & Related papers  (2023-10-18T18:26:01Z)
• Task Formulation Matters When Learning Continually: A Case Study in Visual Question Answering [58.82325933356066]
  Continual learning aims to train a model incrementally on a sequence of tasks without forgetting previous knowledge. We present a detailed study of how different settings affect performance for Visual Question Answering.
  arXiv  Detail & Related papers  (2022-09-30T19:12:58Z)
• Fully Online Meta-Learning Without Task Boundaries [80.09124768759564]
  We study how meta-learning can be applied to tackle online problems of this nature. We propose a Fully Online Meta-Learning (FOML) algorithm, which does not require any ground truth knowledge about the task boundaries. Our experiments show that FOML was able to learn new tasks faster than the state-of-the-art online learning methods.
  arXiv  Detail & Related papers  (2022-02-01T07:51:24Z)
• Multi-Task Learning with Sequence-Conditioned Transporter Networks [67.57293592529517]
  We aim to solve multi-task learning through the lens of sequence-conditioning and weighted sampling. We propose a new suite of benchmark aimed at compositional tasks, MultiRavens, which allows defining custom task combinations. Second, we propose a vision-based end-to-end system architecture, Sequence-Conditioned Transporter Networks, which augments Goal-Conditioned Transporter Networks with sequence-conditioning and weighted sampling.
  arXiv  Detail & Related papers  (2021-09-15T21:19:11Z)
• Visual Adversarial Imitation Learning using Variational Models [60.69745540036375]
  Reward function specification remains a major impediment for learning behaviors through deep reinforcement learning. Visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We develop a variational model-based adversarial imitation learning algorithm.
  arXiv  Detail & Related papers  (2021-07-16T00:15:18Z)
• Lifelong Learning of Few-shot Learners across NLP Tasks [45.273018249235705]
  We study the challenge of lifelong learning to few-shot learn over a sequence of diverse NLP tasks. We propose a continual meta-learning approach which learns to generate adapter weights from a few examples. We demonstrate our approach preserves model performance over training tasks and leads to positive knowledge transfer when the future tasks are learned.
  arXiv  Detail & Related papers  (2021-04-18T10:41:56Z)
• Learning Adaptable Policy via Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks [2.1485350418225244]
  We build an adaptable imitation learning model based on the integration of Meta-learning and Adversarial Inverse Reinforcement Learning. We exploit the adversarial learning and inverse reinforcement learning mechanisms to learn policies and reward functions simultaneously from available training tasks.
  arXiv  Detail & Related papers  (2021-03-23T17:16:38Z)
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning [79.32403825036792]
  We propose a method for pre-training behavioral priors that can capture complex input-output relationships observed in successful trials. We show how this learned prior can be used for rapidly learning new tasks without impeding the RL agent's ability to try out novel behaviors.
  arXiv  Detail & Related papers  (2020-11-19T18:47:40Z)
• Meta Adaptation using Importance Weighted Demonstrations [19.37671674146514]
  In some cases, the distribution shifts, so much, that it is difficult for an agent to infer the new task. We propose a novel algorithm to generalize on any related task by leveraging prior knowledge on a set of specific tasks. We show experiments where the robot is trained from a diversity of environmental tasks and is also able to adapt to an unseen environment.
  arXiv  Detail & Related papers  (2019-11-23T07:22:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.