Augmenting Autotelic Agents with Large Language Models
- URL: http://arxiv.org/abs/2305.12487v1
- Date: Sun, 21 May 2023 15:42:41 GMT
- Title: Augmenting Autotelic Agents with Large Language Models
- Authors: C\'edric Colas, Laetitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan,
Marc-Alexandre C\^ot\'e
- Abstract summary: We introduce a language model augmented autotelic agent (LMA3)
LMA3 supports the representation, generation and learning of diverse, abstract, human-relevant goals.
We show that LMA3 agents learn to master a large diversity of skills in a task-agnostic text-based environment.
- Score: 24.16977502082188
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Humans learn to master open-ended repertoires of skills by imagining and
practicing their own goals. This autotelic learning process, literally the
pursuit of self-generated (auto) goals (telos), becomes more and more
open-ended as the goals become more diverse, abstract and creative. The
resulting exploration of the space of possible skills is supported by an
inter-individual exploration: goal representations are culturally evolved and
transmitted across individuals, in particular using language. Current
artificial agents mostly rely on predefined goal representations corresponding
to goal spaces that are either bounded (e.g. list of instructions), or
unbounded (e.g. the space of possible visual inputs) but are rarely endowed
with the ability to reshape their goal representations, to form new
abstractions or to imagine creative goals. In this paper, we introduce a
language model augmented autotelic agent (LMA3) that leverages a pretrained
language model (LM) to support the representation, generation and learning of
diverse, abstract, human-relevant goals. The LM is used as an imperfect model
of human cultural transmission; an attempt to capture aspects of humans'
common-sense, intuitive physics and overall interests. Specifically, it
supports three key components of the autotelic architecture: 1)~a relabeler
that describes the goals achieved in the agent's trajectories, 2)~a goal
generator that suggests new high-level goals along with their decomposition
into subgoals the agent already masters, and 3)~reward functions for each of
these goals. Without relying on any hand-coded goal representations, reward
functions or curriculum, we show that LMA3 agents learn to master a large
diversity of skills in a task-agnostic text-based environment.
Related papers
- WANDR: Intention-guided Human Motion Generation [67.07028110459787]
We introduce WANDR, a data-driven model that takes an avatar's initial pose and a goal's 3D position and generates natural human motions that place the end effector (wrist) on the goal location.
Intention guides the agent to the goal, and interactively adapts the generation to novel situations without needing to define sub-goals or the entire motion path.
We evaluate our method extensively and demonstrate its ability to generate natural and long-term motions that reach 3D goals and to unseen goal locations.
arXiv Detail & Related papers (2024-04-23T10:20:17Z) - Vision-Language Models as a Source of Rewards [68.52824755339806]
We investigate the feasibility of using off-the-shelf vision-language models, or VLMs, as sources of rewards for reinforcement learning agents.
We show how rewards for visual achievement of a variety of language goals can be derived from the CLIP family of models, and used to train RL agents that can achieve a variety of language goals.
arXiv Detail & Related papers (2023-12-14T18:06:17Z) - A Song of Ice and Fire: Analyzing Textual Autotelic Agents in
ScienceWorld [21.29303927728839]
Building open-ended agents that can autonomously discover a diversity of behaviours is one of the long-standing goals of artificial intelligence.
Recent work identified language has a key dimension of autotelic learning, in particular because it enables abstract goal sampling and guidance from social peers for hindsight relabelling.
We show the importance of selectivity from the social peer's feedback; that experience replay needs to over-sample examples of rare goals.
arXiv Detail & Related papers (2023-02-10T13:49:50Z) - Discrete Factorial Representations as an Abstraction for Goal
Conditioned Reinforcement Learning [99.38163119531745]
We show that applying a discretizing bottleneck can improve performance in goal-conditioned RL setups.
We experimentally prove the expected return on out-of-distribution goals, while still allowing for specifying goals with expressive structure.
arXiv Detail & Related papers (2022-11-01T03:31:43Z) - Vygotskian Autotelic Artificial Intelligence: Language and Culture
Internalization for Human-Like AI [16.487953861478054]
This perspective paper proposes a new AI paradigm in the quest for artificial lifelong skill discovery.
We focus on language especially, and how its structure and content may support the development of new cognitive functions in artificial agents.
It justifies the approach by uncovering examples of new artificial cognitive functions emerging from interactions between language and embodiment.
arXiv Detail & Related papers (2022-06-02T16:35:41Z) - Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning [71.52722621691365]
Building generalizable goal-conditioned agents from rich observations is a key to reinforcement learning (RL) solving real world problems.
We propose a new form of state abstraction called goal-conditioned bisimulation.
We learn this representation using a metric form of this abstraction, and show its ability to generalize to new goals in simulation manipulation tasks.
arXiv Detail & Related papers (2022-04-27T17:00:11Z) - Learning Object-Centered Autotelic Behaviors with Graph Neural Networks [10.149376933379036]
Humans have access to a handful of previously learned skills, which they rapidly adapt to new situations.
In artificial intelligence, autotelic agents, which are intrinsically motivated to represent and set their own goals, exhibit promising skill adaptation capabilities.
We study different implementations of autotelic agents using four types of Graph Neural Networks policy representations and two types of goal spaces.
arXiv Detail & Related papers (2022-04-11T14:19:04Z) - Learning with AMIGo: Adversarially Motivated Intrinsic Goals [63.680207855344875]
AMIGo is a goal-generating teacher that proposes Adversarially Motivated Intrinsic Goals.
We show that our method generates a natural curriculum of self-proposed goals which ultimately allows the agent to solve challenging procedurally-generated tasks.
arXiv Detail & Related papers (2020-06-22T10:22:08Z) - Automatic Curriculum Learning through Value Disagreement [95.19299356298876]
Continually solving new, unsolved tasks is the key to learning diverse behaviors.
In the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency.
We propose setting up an automatic curriculum for goals that the agent needs to solve.
We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.
arXiv Detail & Related papers (2020-06-17T03:58:25Z) - Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven
Exploration [15.255795563999422]
Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills.
We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning.
We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity.
arXiv Detail & Related papers (2020-02-21T12:59:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.