GUI Agents for Continual Game Generation

• URL: http://arxiv.org/abs/2605.28258v1
• Date: Wed, 27 May 2026 10:08:48 GMT
• Title: GUI Agents for Continual Game Generation
• Abstract summary: We argue that evaluating and improving game generation requires a player.<n>We introduce PlaytestArena, a new evaluation environment that pairs 200 browser-based game generation tasks with rubrics of expected in-play behaviors.<n>We propose Play2Code, where a game agent and a GUI agent operate in a sustained loop with shared memory, turning game generation into a dialogue between coding and playing.
• Score: 22.25472214301186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating a game is not the same as making one that can be played. Despite advances in code generation, existing approaches treat game generation as one-shot translation from prompt to artifact, leaving interaction-level failures undetected. We argue that evaluating and improving game generation requires a player, and study two roles for graphical user interface (GUI) agents in this process: (1) as an objective evaluator, for which we introduce PlaytestArena, a new evaluation environment that pairs 200 browser-based game generation tasks across eight genres with rubrics of expected in-play behaviors, adjudicated by a GUI agent that loads each build in a browser and plays it; and (2) as a subjective playtester, for which we propose Play2Code, where a game agent and a GUI agent operate in a sustained loop with shared memory, turning game generation into a dialogue between coding and playing. Our experiments show that even frontier models struggle to generate playable games directly, while Play2Code achieves a 66.8\% rubric pass-rate, improving over single-pass and agentic-coding baselines by 37.1 and 14.6 points respectively. Further analysis shows that GUI playtester feedback is more traceable than a human report, yet idiosyncratic in ways reminiscent of human testers, establishing game playtesting as a critical testbed for interactive code generation. Our project website is available at https://continual-game-generation.vercel.app/.

Related papers

• GameXpert-Bench: How Far Are Coding Agents from Expert Game Development? [6.806647989372878]
  Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests.<n>Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact.<n>Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent.
  arXiv  Detail & Related papers  (2026-08-22T08:05:16Z)
• GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine? [65.42976417627254]
  Game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay.<n>We formalize end-to-end game generation as the problem of producing a complete game artifact.<n>We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families.
  arXiv  Detail & Related papers  (2026-06-16T12:34:39Z)
• PlayCoder: Making LLM-Generated GUI Code Playable [30.115742334937977]
  Existing benchmarks mainly evaluate correctness through test cases, which are inadequate for GUI applications.<n>We introduce PlayEval, a repository-aware benchmark built from 43 multilingual GUI applications in Python, TypeScript, and JavaScript.<n>We also propose Play@k, a metric that measures whether at least one of *k* generated candidates can be played end-to-end without logical errors.
  arXiv  Detail & Related papers  (2026-04-21T17:59:16Z)
• GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents [76.60994803070436]
  GameWorld is a benchmark for evaluation of Multimodal Large Language Model (MLLM) game agents in browser environments.<n>Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space.<n>Results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games.
  arXiv  Detail & Related papers  (2026-04-08T17:49:03Z)
• GameDevBench: Evaluating Agentic Capabilities Through Game Development [49.19956546746812]
  Game development provides such a testbed as agents must navigate large, denses while manipulating intrinsically multimodal assets.<n>We present GameDevBench, the first benchmark for evaluating agents on game development tasks.<n>Agents still struggle with game development, with the best agent solving only 54.5% of tasks.
  arXiv  Detail & Related papers  (2026-02-11T18:15:11Z)
• FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback [92.67587639164908]
  We present FronTalk, a benchmark for front-end code generation with multi-modal feedback.<n>We focus on the front-end development task and curate FronTalk, a collection of 100 multi-turn dialogues.<n> Evaluation of 20 models reveals two key challenges that are under-explored systematically in the literature.
  arXiv  Detail & Related papers  (2025-12-05T23:28:09Z)
• FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Games [56.81554611870848]
  We introduce FlashAdventure, a benchmark of 34 Flash-based adventure games designed to test full story arc completion.<n>We also propose CUA-as-a-Judge, an automated gameplay evaluator, and COAST, an agentic framework leveraging long-term clue memory.<n> Experiments show current GUI agents struggle with full story arcs, while COAST improves milestone completion by bridging the observation-behavior gap.
  arXiv  Detail & Related papers  (2025-09-01T01:33:16Z)
• Pixels to Play: A Foundation Model for 3D Gameplay [4.380638021267298]
  We introduce Pixels2Play-0.1 (P2P0.1), a foundation model that learns to play a wide range of 3D video games with recognizable human-like behavior.
  arXiv  Detail & Related papers  (2025-08-19T22:24:50Z)
• Playable Game Generation [22.17100581717806]
  We propose emphPlayGen, which encompasses game data generation, an autoregressive DiT-based diffusion model, and a playability-based evaluation framework.<n>PlayGen achieves real-time interaction, ensures sufficient visual quality, and provides accurate interactive mechanics simulation.
  arXiv  Detail & Related papers  (2024-12-01T16:53:02Z)
• PlayTest: A Gamified Test Generator for Games [11.077232808482128]
  Playtest transforms the tiring testing process into a competitive game with a purpose. We envision the use of Playtest to crowdsource the task of testing games by giving players access to the respective games through our tool in the playtesting phases.
  arXiv  Detail & Related papers  (2023-10-30T10:14:27Z)
• Inspector: Pixel-Based Automated Game Testing via Exploration, Detection, and Investigation [116.41186277555386]
  Inspector is a game testing agent that can be easily applied to different games without deep integration with games. Inspector is based on purely pixel inputs and comprises three key modules: game space explorer, key object detector, and human-like object investigator. Experiment results demonstrate the effectiveness of Inspector in exploring game space, detecting key objects, and investigating objects.
  arXiv  Detail & Related papers  (2022-07-18T04:49:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.