Towards LLM-Based Automatic Playtest

• URL: http://arxiv.org/abs/2507.09490v1
• Date: Sun, 13 Jul 2025 04:23:44 GMT
• Title: Towards LLM-Based Automatic Playtest
• Authors: Yan Zhao, Chiwei Tang,
• Abstract summary: Playtesting is critical for the quality assurance of gaming software.<n>Recent advancements in artificial intelligence (AI) have opened up new possibilities for applying Large Language Models (LLMs) to playtesting.<n>This paper introduces Lap, our novel approach to Automatic Playtesting.
• Score: 1.9714447272714082
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Playtesting is the process in which people play a video game for testing. It is critical for the quality assurance of gaming software. Manual playtesting is time-consuming and expensive. However, automating this process is challenging, as playtesting typically requires domain knowledge and problem-solving skills that most conventional testing tools lack. Recent advancements in artificial intelligence (AI) have opened up new possibilities for applying Large Language Models (LLMs) to playtesting. However, significant challenges remain: current LLMs cannot visually perceive game environments, and most existing research focuses on text-based games or games with robust APIs. Many non-text games lack APIs to provide textual descriptions of game states, making it almost impossible to naively apply LLMs for playtesting. This paper introduces Lap, our novel approach to LLM-based Automatic Playtesting, which uses ChatGPT to test match-3 games, a category of games where players match three or more identical tiles in a row or column to earn points. Lap encompasses three key phases: processing of game environments, prompting-based action generation, and action execution. Given a match-3 game, Lap takes a snapshot of the game board and converts it to a numeric matrix. It then prompts the ChatGPT-O1-mini API to suggest moves based on that matrix and tentatively applies the suggested moves to earn points and trigger changes in the game board. It repeats the above-mentioned three steps iteratively until timeout. For evaluation, we conducted a case study using Lap on an open-source match-3 game, CasseBonbons, and empirically compared it with three existing tools. Our results are promising: Lap outperformed existing tools by achieving higher code coverage and triggering more program crashes. This research sheds light on the future of automatic testing and LLM applications.

Related papers

• Who is a Better Player: LLM against LLM [53.46608216197315]
  We propose an adversarial benchmarking framework to assess the comprehensive performance of Large Language Models (LLMs) through board games competition.<n>We introduce Qi Town, a specialized evaluation platform that supports 5 widely played games and involves 20 LLM-driven players.
  arXiv  Detail & Related papers  (2025-08-05T06:41:47Z)
• lmgame-Bench: How Good are LLMs at Playing Games? [60.01834131847881]
  We study the major challenges in using popular video games to evaluate modern large language model (LLM) agents.<n>We introduce lmgame-Bench to turn games into reliable evaluations.
  arXiv  Detail & Related papers  (2025-05-21T06:02:55Z)
• GAMEBoT: Transparent Assessment of LLM Reasoning in Games [54.49589494014147]
  GAMEBoT is a gaming arena designed for rigorous assessment of Large Language Models.<n>We benchmark 17 prominent LLMs across eight games, encompassing various strategic abilities and game characteristics.<n>Our results suggest that GAMEBoT presents a significant challenge, even when LLMs are provided with detailed CoT prompts.
  arXiv  Detail & Related papers  (2024-12-18T08:32:53Z)
• LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection [87.43727192273772]
  It is often hard to tell whether a piece of text was human-written or machine-generated.<n>We present LLM-DetectAIve, designed for fine-grained detection.<n>It supports four categories: (i) human-written, (ii) machine-generated, (iii) machine-written, then machine-humanized, and (iv) human-written, then machine-polished.
  arXiv  Detail & Related papers  (2024-08-08T07:43:17Z)
• Cooperative Multi-agent Approach for Automated Computer Game Testing [1.4931265249949526]
  Many games nowadays are multi-player. This opens up an interesting possibility to deploy multiple cooperative test agents to test such a game. This paper offers a cooperative multi-agent testing approach and a study of its performance based on a case study on a 3D game called Lab Recruits.
  arXiv  Detail & Related papers  (2024-05-18T17:31:26Z)
• PlayTest: A Gamified Test Generator for Games [11.077232808482128]
  Playtest transforms the tiring testing process into a competitive game with a purpose. We envision the use of Playtest to crowdsource the task of testing games by giving players access to the respective games through our tool in the playtesting phases.
  arXiv  Detail & Related papers  (2023-10-30T10:14:27Z)
• SmartPlay: A Benchmark for LLMs as Intelligent Agents [45.76707302899935]
  SmartPlay consists of 6 different games, including Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game challenges a subset of 9 important capabilities of an intelligent LLM agent. Tests include reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness.
  arXiv  Detail & Related papers  (2023-10-02T18:52:11Z)
• SPRING: Studying the Paper and Reasoning to Play Games [102.5587155284795]
  We propose a novel approach, SPRING, to read the game's original academic paper and use the knowledge learned to reason and play the game through a large language model (LLM) In experiments, we study the quality of in-context "reasoning" induced by different forms of prompts under the setting of the Crafter open-world environment. Our experiments suggest that LLMs, when prompted with consistent chain-of-thought, have great potential in completing sophisticated high-level trajectories.
  arXiv  Detail & Related papers  (2023-05-24T18:14:35Z)
• Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing [23.460051600514806]
  We propose GPTDroid, asking Large Language Model to chat with the mobile apps by passing the GUI page information to LLM to elicit testing scripts. Within it, we extract the static context of the GUI page and the dynamic context of the iterative testing process. We evaluate GPTDroid on 86 apps from Google Play, and its activity coverage is 71%, with 32% higher than the best baseline, and can detect 36% more bugs with faster speed than the best baseline.
  arXiv  Detail & Related papers  (2023-05-16T13:46:52Z)
• An Empirical Study on the Generalization Power of Neural Representations Learned via Visual Guessing Games [79.23847247132345]
  This work investigates how well an artificial agent can benefit from playing guessing games when later asked to perform on novel NLP downstream tasks such as Visual Question Answering (VQA) We propose two ways to exploit playing guessing games: 1) a supervised learning scenario in which the agent learns to mimic successful guessing games and 2) a novel way for an agent to play by itself, called Self-play via Iterated Experience Learning (SPIEL)
  arXiv  Detail & Related papers  (2021-01-31T10:30:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.