GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?
- URL: http://arxiv.org/abs/2608.21833v1
- Date: Sat, 22 Aug 2026 08:05:16 GMT
- Title: GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?
- Abstract summary: Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests.<n>Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact.<n>Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent.
- Score: 6.806647989372878
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact. Measuring this capability therefore requires evaluation of both game product and the development process. Existing benchmarks often assess the game development capabilities of LLMs by evaluating the final artifact or an isolated development stage. Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent: initial game generation, bug diagnosis and repair, and optimization over multiple turns. Therefore, we introduce GameXpert-Bench, which operationalizes the three lifecycle stages as three complementary benchmark tracks. GameGen evaluates complete game creation from a single request in an empty workspace. GameFix evaluates diagnosis and repair when defects are reported or left for the agent to discover. GameOpt evaluates cumulative optimization through request chains seeded by real development trajectories between users and agents. We evaluate each track using live game interaction, deterministic behavioral tests, or final product criteria with regression checks. The suite contains 97 generation tasks across 11 genres; 100 repair tasks from 50 game levels verified by humans, each with 19-27 injected bugs; and 17 optimization chains with six turns and 102 requests. Across the three tracks, current agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes.
Related papers
- GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments [1.392548092257887]
We present GameEngineBench, a benchmark for evaluating coding agents on scoped C++ implementation tasks inside Unreal Engine 5 projects.<n>The evaluation set consists of 110 tasks spanning gameplay mechanics, multiplayer behavior, AI and world orchestration, animation and movement, UI and session code, loading behavior, online-service integration, persistence, data serialization, XR behavior, and rendering-oriented plugins.
arXiv Detail & Related papers (2026-07-03T17:56:26Z) - GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine? [65.42976417627254]
Game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay.<n>We formalize end-to-end game generation as the problem of producing a complete game artifact.<n>We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families.
arXiv Detail & Related papers (2026-06-16T12:34:39Z) - GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents [76.60994803070436]
GameWorld is a benchmark for evaluation of Multimodal Large Language Model (MLLM) game agents in browser environments.<n>Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space.<n>Results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games.
arXiv Detail & Related papers (2026-04-08T17:49:03Z) - ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
arXiv Detail & Related papers (2026-04-05T17:55:23Z) - GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers [1.6086198362375759]
We introduce the Game Benchmark for Quality Assurance (GBQA) to evaluate whether large language models can autonomously detect software bugs.<n>The benchmark is constructed using a multi-agent system that develops games and injects bugs in a scalable manner, with human experts in the loop to ensure correctness.<n>In experiments, the best-performing model, Claude-4.6-Opus in thinking mode, identifies only 48.39% of the verified bugs.
arXiv Detail & Related papers (2026-04-03T02:23:50Z) - SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
arXiv Detail & Related papers (2026-03-25T19:26:44Z) - GameDevBench: Evaluating Agentic Capabilities Through Game Development [49.19956546746812]
Game development provides such a testbed as agents must navigate large, denses while manipulating intrinsically multimodal assets.<n>We present GameDevBench, the first benchmark for evaluating agents on game development tasks.<n>Agents still struggle with game development, with the best agent solving only 54.5% of tasks.
arXiv Detail & Related papers (2026-02-11T18:15:11Z) - FlashAdventure: A Benchmark for GUI Agents Solving Full Story Arcs in Diverse Adventure Games [56.81554611870848]
We introduce FlashAdventure, a benchmark of 34 Flash-based adventure games designed to test full story arc completion.<n>We also propose CUA-as-a-Judge, an automated gameplay evaluator, and COAST, an agentic framework leveraging long-term clue memory.<n> Experiments show current GUI agents struggle with full story arcs, while COAST improves milestone completion by bridging the observation-behavior gap.
arXiv Detail & Related papers (2025-09-01T01:33:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.