Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case

• URL: http://arxiv.org/abs/2409.12889v2
• Date: Sun, 22 Sep 2024 09:51:58 GMT
• Title: Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case
• Authors: Peng Chen, Pi Bu, Jun Song, Yuan Gao, Bo Zheng,
• Abstract summary: This research aims to provide new insights and directions for applying multimodal agents in complex action game environments. We select an ARPG, Black Myth: Wukong'', as a research platform to explore the capability boundaries of existing vision language models. We will release a human operation dataset containing recorded gameplay videos and operation logs, including mouse and keyboard actions.
• Score: 20.14197375326218
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, large language model (LLM)-based agents have made significant advances across various fields. One of the most popular research areas involves applying these agents to video games. Traditionally, these methods have relied on game APIs to access in-game environmental and action data. However, this approach is limited by the availability of APIs and does not reflect how humans play games. With the advent of vision language models (VLMs), agents now have enhanced visual understanding capabilities, enabling them to interact with games using only visual inputs. Despite these advances, current approaches still face challenges in action-oriented tasks, particularly in action role-playing games (ARPGs), where reinforcement learning methods are prevalent but suffer from poor generalization and require extensive training. To address these limitations, we select an ARPG, ``Black Myth: Wukong'', as a research platform to explore the capability boundaries of existing VLMs in scenarios requiring visual-only input and complex action output. We define 12 tasks within the game, with 75% focusing on combat, and incorporate several state-of-the-art VLMs into this benchmark. Additionally, we will release a human operation dataset containing recorded gameplay videos and operation logs, including mouse and keyboard actions. Moreover, we propose a novel VARP (Vision Action Role-Playing) agent framework, consisting of an action planning system and a visual trajectory system. Our framework demonstrates the ability to perform basic tasks and succeed in 90% of easy and medium-level combat scenarios. This research aims to provide new insights and directions for applying multimodal agents in complex action game environments. The code and datasets will be made available at https://varp-agent.github.io/.

Related papers

• Atari-GPT: Investigating the Capabilities of Multimodal Large Language Models as Low-Level Policies for Atari Games [2.2648566044372416]
  This paper explores the application of multimodal large language models (LLMs) as low-level controllers in the domain of Atari video games. Unlike traditional reinforcement learning (RL) and imitation learning (IL) methods, these LLMs utilize pre-existing multimodal knowledge to directly engage with game environments.
  arXiv  Detail & Related papers  (2024-08-28T17:08:56Z)
• A Survey on Large Language Model-Based Game Agents [9.892954815419452]
  The development of game agents holds a critical role in advancing towards Artificial General Intelligence (AGI) This paper provides a comprehensive overview of LLM-based game agents from a holistic viewpoint.
  arXiv  Detail & Related papers  (2024-04-02T15:34:18Z)
• Scaling Instructable Agents Across Many Simulated Worlds [70.97268311053328]
  Our goal is to develop an agent that can accomplish anything a human can do in any simulated 3D environment. Our approach focuses on language-driven generality while imposing minimal assumptions. Our agents interact with environments in real-time using a generic, human-like interface.
  arXiv  Detail & Related papers  (2024-03-13T17:50:32Z)
• Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games [26.07074182316433]
  We introduce the first dataset specifically for Jubensha, including character scripts and game rules. Our work also presents a unique multi-agent interaction framework using LLMs, allowing AI agents to autonomously engage in this game. To evaluate the gaming performance of these AI agents, we developed novel methods measuring their mastery of case information and reasoning skills.
  arXiv  Detail & Related papers  (2023-12-01T17:33:57Z)
• A Minimal Approach for Natural Language Action Space in Text-based Games [103.21433712630953]
  This paper revisits the challenge of exploring the action space in text-based games (TGs) We propose $ epsilon$-admissible exploration, a minimal approach of utilizing admissible actions, for training phase. We present a text-based actor-critic (TAC) agent that produces textual commands for game, solely from game observations.
  arXiv  Detail & Related papers  (2023-05-06T16:05:27Z)
• Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models [68.85478477006178]
  We present a Promptable Game Model (PGM) for neural video game simulators. It allows a user to play the game by prompting it with high- and low-level action sequences. Most captivatingly, our PGM unlocks the director's mode, where the game is played by specifying goals for the agents in the form of a prompt. Our method significantly outperforms existing neural video game simulators in terms of rendering quality and unlocks applications beyond the capabilities of the current state of the art.
  arXiv  Detail & Related papers  (2023-03-23T17:43:17Z)
• Learning to Simulate Dynamic Environments with GameGAN [109.25308647431952]
  In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training.
  arXiv  Detail & Related papers  (2020-05-25T14:10:17Z)
• Neural MMO v1.3: A Massively Multiagent Game Environment for Training and Evaluating Neural Networks [48.5733173329785]
  We present Neural MMO, a massively multiagent game environment inspired by MMOs. We discuss our progress on two more general challenges in multiagent systems engineering for AI research: distributed infrastructure and game IO.
  arXiv  Detail & Related papers  (2020-01-31T18:50:02Z)
• Exploration Based Language Learning for Text-Based Games [72.30525050367216]
  This work presents an exploration and imitation-learning-based agent capable of state-of-the-art performance in playing text-based computer games. Text-based computer games describe their world to the player through natural language and expect the player to interact with the game using text. These games are of interest as they can be seen as a testbed for language understanding, problem-solving, and language generation by artificial agents.
  arXiv  Detail & Related papers  (2020-01-24T03:03:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.