Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration

• URL: http://arxiv.org/abs/2002.09253v4
• Date: Wed, 21 Oct 2020 16:48:51 GMT
• Title: Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Authors: C\'edric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Cl\'ement Moulin-Frier, Peter Ford Dominey, Pierre-Yves Oudeyer
• Abstract summary: Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity.
• Score: 15.255795563999422
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. Children do so by leveraging the compositionality of language as a tool to imagine descriptions of outcomes they never experienced before, targeting them as goals during play. We introduce IMAGINE, an intrinsically motivated deep reinforcement learning architecture that models this ability. Such imaginative agents, like children, benefit from the guidance of a social peer who provides language descriptions. To take advantage of goal imagination, agents must be able to leverage these descriptions to interpret their imagined out-of-distribution goals. This generalization is made possible by modularity: a decomposition between learned goal-achievement reward function and policy relying on deep sets, gated attention and object-centered representations. We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity. In addition, we identify the properties of goal imagination that enable these results and study the impacts of modularity and social interactions.

Related papers

• Vision-Language Models as a Source of Rewards [68.52824755339806]
  We investigate the feasibility of using off-the-shelf vision-language models, or VLMs, as sources of rewards for reinforcement learning agents. We show how rewards for visual achievement of a variety of language goals can be derived from the CLIP family of models, and used to train RL agents that can achieve a variety of language goals.
  arXiv  Detail & Related papers  (2023-12-14T18:06:17Z)
• Learning to Model the World with Language [100.76069091703505]
  To interact with humans and act in the world, agents need to understand the range of language that people use and relate it to the visual world. Our key idea is that agents should interpret such diverse language as a signal that helps them predict the future. We instantiate this in Dynalang, an agent that learns a multimodal world model to predict future text and image representations.
  arXiv  Detail & Related papers  (2023-07-31T17:57:49Z)
• Augmenting Autotelic Agents with Large Language Models [24.16977502082188]
  We introduce a language model augmented autotelic agent (LMA3) LMA3 supports the representation, generation and learning of diverse, abstract, human-relevant goals. We show that LMA3 agents learn to master a large diversity of skills in a task-agnostic text-based environment.
  arXiv  Detail & Related papers  (2023-05-21T15:42:41Z)
• A Song of Ice and Fire: Analyzing Textual Autotelic Agents in ScienceWorld [21.29303927728839]
  Building open-ended agents that can autonomously discover a diversity of behaviours is one of the long-standing goals of artificial intelligence. Recent work identified language has a key dimension of autotelic learning, in particular because it enables abstract goal sampling and guidance from social peers for hindsight relabelling. We show the importance of selectivity from the social peer's feedback; that experience replay needs to over-sample examples of rare goals.
  arXiv  Detail & Related papers  (2023-02-10T13:49:50Z)
• Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning [71.52722621691365]
  Building generalizable goal-conditioned agents from rich observations is a key to reinforcement learning (RL) solving real world problems. We propose a new form of state abstraction called goal-conditioned bisimulation. We learn this representation using a metric form of this abstraction, and show its ability to generalize to new goals in simulation manipulation tasks.
  arXiv  Detail & Related papers  (2022-04-27T17:00:11Z)
• Understanding the origin of information-seeking exploration in probabilistic objectives for control [62.997667081978825]
  An exploration-exploitation trade-off is central to the description of adaptive behaviour. One approach to solving this trade-off has been to equip or propose that agents possess an intrinsic 'exploratory drive' We show that this combination of utility maximizing and information-seeking behaviour arises from the minimization of an entirely difference class of objectives.
  arXiv  Detail & Related papers  (2021-03-11T18:42:39Z)
• Action and Perception as Divergence Minimization [43.75550755678525]
  Action Perception Divergence is an approach for categorizing the space of possible objective functions for embodied agents. We show a spectrum that reaches from narrow to general objectives. These agents use perception to align their beliefs with the world and use actions to align the world with their beliefs.
  arXiv  Detail & Related papers  (2020-09-03T16:52:46Z)
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning [21.661530291654692]
  We propose a framework that allows agents to autonomously identify and ignore noisy distracting regions. Our framework can be combined with any state-of-the-art novelty seeking goal exploration approaches.
  arXiv  Detail & Related papers  (2020-08-10T19:50:06Z)
• Learning with AMIGo: Adversarially Motivated Intrinsic Goals [63.680207855344875]
  AMIGo is a goal-generating teacher that proposes Adversarially Motivated Intrinsic Goals. We show that our method generates a natural curriculum of self-proposed goals which ultimately allows the agent to solve challenging procedurally-generated tasks.
  arXiv  Detail & Related papers  (2020-06-22T10:22:08Z)
• Imagination-Augmented Deep Learning for Goal Recognition [0.0]
  A prominent idea in current goal-recognition research is to infer the likelihood of an agent's goal from the estimations of the costs of plans to the different goals the agent might have. This paper introduces a novel idea of using a symbolic planner to compute plan-cost insights, which augment a deep neural network with an imagination capability.
  arXiv  Detail & Related papers  (2020-03-20T23:07:34Z)
• Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning [102.05692309417047]
  In reinforcement learning, an agent learns to reach a set of goals by means of an external reward signal. In the natural world, intelligent organisms learn from internal drives, bypassing the need for external signals. We propose to formulate an intrinsic objective as the mutual information between the goal states and the controllable states.
  arXiv  Detail & Related papers  (2020-02-05T19:21:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.