A Song of Ice and Fire: Analyzing Textual Autotelic Agents in ScienceWorld

• URL: http://arxiv.org/abs/2302.05244v3
• Date: Tue, 14 Feb 2023 11:20:52 GMT
• Title: A Song of Ice and Fire: Analyzing Textual Autotelic Agents in ScienceWorld
• Authors: Laetitia Teodorescu, Eric Yuan, Marc-Alexandre C\^ot\'e, Pierre-Yves Oudeyer
• Abstract summary: Building open-ended agents that can autonomously discover a diversity of behaviours is one of the long-standing goals of artificial intelligence. Recent work identified language has a key dimension of autotelic learning, in particular because it enables abstract goal sampling and guidance from social peers for hindsight relabelling. We show the importance of selectivity from the social peer's feedback; that experience replay needs to over-sample examples of rare goals.
• Score: 21.29303927728839
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Building open-ended agents that can autonomously discover a diversity of behaviours is one of the long-standing goals of artificial intelligence. This challenge can be studied in the framework of autotelic RL agents, i.e. agents that learn by selecting and pursuing their own goals, self-organizing a learning curriculum. Recent work identified language has a key dimension of autotelic learning, in particular because it enables abstract goal sampling and guidance from social peers for hindsight relabelling. Within this perspective, we study the following open scientific questions: What is the impact of hindsight feedback from a social peer (e.g. selective vs. exhaustive)? How can the agent learn from very rare language goal examples in its experience replay? How can multiple forms of exploration be combined, and take advantage of easier goals as stepping stones to reach harder ones? To address these questions, we use ScienceWorld, a textual environment with rich abstract and combinatorial physics. We show the importance of selectivity from the social peer's feedback; that experience replay needs to over-sample examples of rare goals; and that following self-generated goal sequences where the agent's competence is intermediate leads to significant improvements in final performance.

Related papers

• AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios [38.878966229688054]
  We introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning.
  arXiv  Detail & Related papers  (2024-10-25T07:04:16Z)
• Augmenting Autotelic Agents with Large Language Models [24.16977502082188]
  We introduce a language model augmented autotelic agent (LMA3) LMA3 supports the representation, generation and learning of diverse, abstract, human-relevant goals. We show that LMA3 agents learn to master a large diversity of skills in a task-agnostic text-based environment.
  arXiv  Detail & Related papers  (2023-05-21T15:42:41Z)
• EgoTaskQA: Understanding Human Tasks in Egocentric Videos [89.9573084127155]
  EgoTaskQA benchmark provides home for crucial dimensions of task understanding through question-answering on real-world egocentric videos. We meticulously design questions that target the understanding of (1) action dependencies and effects, (2) intents and goals, and (3) agents' beliefs about others. We evaluate state-of-the-art video reasoning models on our benchmark and show their significant gaps between humans in understanding complex goal-oriented egocentric videos.
  arXiv  Detail & Related papers  (2022-10-08T05:49:05Z)
• Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning [71.52722621691365]
  Building generalizable goal-conditioned agents from rich observations is a key to reinforcement learning (RL) solving real world problems. We propose a new form of state abstraction called goal-conditioned bisimulation. We learn this representation using a metric form of this abstraction, and show its ability to generalize to new goals in simulation manipulation tasks.
  arXiv  Detail & Related papers  (2022-04-27T17:00:11Z)
• Help Me Explore: Minimal Social Interventions for Graph-Based Autotelic Agents [7.644107117422287]
  This paper argues that both perspectives could be coupled within the learning of autotelic agents to foster their skill acquisition. We make two contributions: 1) a novel social interaction protocol called Help Me Explore (HME), where autotelic agents can benefit from both individual and socially guided exploration. We show that when learning within HME, GANGSTR overcomes its individual learning limits by mastering the most complex configurations.
  arXiv  Detail & Related papers  (2022-02-10T16:34:28Z)
• Lipschitz-constrained Unsupervised Skill Discovery [91.51219447057817]
  Lipschitz-constrained Skill Discovery (LSD) encourages the agent to discover more diverse, dynamic, and far-reaching skills. LSD outperforms previous approaches in terms of skill diversity, state space coverage, and performance on seven downstream tasks.
  arXiv  Detail & Related papers  (2022-02-02T08:29:04Z)
• Discovering and Achieving Goals via World Models [61.95437238374288]
  We introduce Latent Explorer Achiever (LEXA), a unified solution to this problem. LEXA learns a world model from image inputs and uses it to train an explorer and an achiever policy from imagined rollouts. After the unsupervised phase, LEXA solves tasks specified as goal images zero-shot without any additional learning.
  arXiv  Detail & Related papers  (2021-10-18T17:59:58Z)
• GRIMGEP: Learning Progress for Robust Goal Sampling in Visual Deep Reinforcement Learning [21.661530291654692]
  We propose a framework that allows agents to autonomously identify and ignore noisy distracting regions. Our framework can be combined with any state-of-the-art novelty seeking goal exploration approaches.
  arXiv  Detail & Related papers  (2020-08-10T19:50:06Z)
• Automatic Curriculum Learning through Value Disagreement [95.19299356298876]
  Continually solving new, unsolved tasks is the key to learning diverse behaviors. In the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. We propose setting up an automatic curriculum for goals that the agent needs to solve. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-17T03:58:25Z)
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration [15.255795563999422]
  Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity.
  arXiv  Detail & Related papers  (2020-02-21T12:59:57Z)
• Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning [102.05692309417047]
  In reinforcement learning, an agent learns to reach a set of goals by means of an external reward signal. In the natural world, intelligent organisms learn from internal drives, bypassing the need for external signals. We propose to formulate an intrinsic objective as the mutual information between the goal states and the controllable states.
  arXiv  Detail & Related papers  (2020-02-05T19:21:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.