TALES: Text Adventure Learning Environment Suite

• URL: http://arxiv.org/abs/2504.14128v4
• Date: Thu, 24 Apr 2025 02:50:28 GMT
• Title: TALES: Text Adventure Learning Environment Suite
• Authors: Christopher Zhang Cui, Xingdi Yuan, Ziang Xiao, Prithviraj Ammanabrolu, Marc-Alexandre Côté,
• Abstract summary: Reasoning is an essential skill to enable Large Language Models (LLMs) to interact with the world.<n>We introduce TALES, a diverse collection of synthetic and human-written text-adventure games designed to challenge and evaluate diverse reasoning capabilities.<n>Despite an impressive showing on synthetic games, even the top LLM-driven agents fail to achieve 15% on games designed for human enjoyment.
• Score: 28.997169350434795
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning is an essential skill to enable Large Language Models (LLMs) to interact with the world. As tasks become more complex, they demand increasingly sophisticated and diverse reasoning capabilities for sequential decision-making, requiring structured reasoning over the context history to determine the next best action. We introduce TALES, a diverse collection of synthetic and human-written text-adventure games designed to challenge and evaluate diverse reasoning capabilities. We present results over a range of LLMs, open- and closed-weights, performing a qualitative analysis on the top performing models. Despite an impressive showing on synthetic games, even the top LLM-driven agents fail to achieve 15% on games designed for human enjoyment. Code and visualization of the experiments can be found at https://microsoft.github.io/tale-suite.

Related papers

• Beyond Outcomes: Transparent Assessment of LLM Reasoning in Games [54.49589494014147]
  GAMEBoT is a gaming arena designed for rigorous assessment of Large Language Models.<n>We benchmark 17 prominent LLMs across eight games, encompassing various strategic abilities and game characteristics.<n>Our results suggest that GAMEBoT presents a significant challenge, even when LLMs are provided with detailed CoT prompts.
  arXiv  Detail & Related papers  (2024-12-18T08:32:53Z)
• GameArena: Evaluating LLM Reasoning through Live Computer Games [25.415321902887598]
  We introduce GameArena, a benchmark to evaluate large language models (LLMs) reasoning capabilities through interactive gameplay with humans. GameArena consists of three games to test specific reasoning capabilities (e.g., deductive and inductive reasoning) while keeping participants entertained and engaged. We collect over 2000 game sessions and provide detailed assessments of various reasoning capabilities for five state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2024-12-09T11:22:59Z)
• Mastering Board Games by External and Internal Planning with Language Models [30.782334791241556]
  We show that search-based planning can yield significant improvements in Large Language Models game-playing strength. We introduce, compare and contrast two major approaches: in external search, the model guides Monte Carlo Tree Search rollouts and evaluations without calls to an external game engine, and in internal search, the model is trained to generate in-context a linearized tree of search and a resulting final choice. Our proposed approach, combining search with domain knowledge, is not specific to board games, hinting at more general future applications.
  arXiv  Detail & Related papers  (2024-12-02T18:56:51Z)
• Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash [6.65572931991284]
  Large Language Models (LLMs) have shown impressive capabilities in complex tasks and interactive environments. This paper introduces a simulation framework utilizing the game Balderdash to evaluate both the creativity and logical reasoning of LLMs.
  arXiv  Detail & Related papers  (2024-11-15T18:42:48Z)
• AMONGAGENTS: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Game [12.384945632524424]
  This paper focuses on creating proxies of human behavior in simulated environments, with Among Us utilized as a tool for studying simulated human behavior. Our work demonstrates that state-of-the-art large language models (LLMs) can effectively grasp the game rules and make decisions based on the current context.
  arXiv  Detail & Related papers  (2024-07-23T14:34:38Z)
• GameEval: Evaluating LLMs on Conversational Games [93.40433639746331]
  We propose GameEval, a novel approach to evaluating large language models (LLMs) GameEval treats LLMs as game players and assigns them distinct roles with specific goals achieved by launching conversations of various forms. We show that GameEval can effectively differentiate the capabilities of various LLMs, providing a comprehensive assessment of their integrated abilities to solve complex problems.
  arXiv  Detail & Related papers  (2023-08-19T14:33:40Z)
• SPRING: Studying the Paper and Reasoning to Play Games [102.5587155284795]
  We propose a novel approach, SPRING, to read the game's original academic paper and use the knowledge learned to reason and play the game through a large language model (LLM) In experiments, we study the quality of in-context "reasoning" induced by different forms of prompts under the setting of the Crafter open-world environment. Our experiments suggest that LLMs, when prompted with consistent chain-of-thought, have great potential in completing sophisticated high-level trajectories.
  arXiv  Detail & Related papers  (2023-05-24T18:14:35Z)
• JECC: Commonsense Reasoning Tasks Derived from Interactive Fictions [75.42526766746515]
  We propose a new commonsense reasoning dataset based on human's Interactive Fiction (IF) gameplay walkthroughs. Our dataset focuses on the assessment of functional commonsense knowledge rules rather than factual knowledge. Experiments show that the introduced dataset is challenging to previous machine reading models as well as the new large language models.
  arXiv  Detail & Related papers  (2022-10-18T19:20:53Z)
• Inner Monologue: Embodied Reasoning through Planning with Language Models [81.07216635735571]
  Large Language Models (LLMs) can be applied to domains beyond natural language processing. LLMs planning in embodied environments need to consider not just what skills to do, but also how and when to do them. We propose that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan in robotic control scenarios.
  arXiv  Detail & Related papers  (2022-07-12T15:20:48Z)
• Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning [94.50608198582636]
  Interactive Fiction (IF) games with real human-written natural language texts provide a new natural evaluation for language understanding techniques. We take a novel perspective of IF game solving and re-formulate it as Multi-Passage Reading (MPRC) tasks.
  arXiv  Detail & Related papers  (2020-10-05T23:09:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.