JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines

• URL: http://arxiv.org/abs/2606.19830v2
• Date: Sun, 21 Jun 2026 05:48:00 GMT
• Title: JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines
• Abstract summary: We present JamSet and JamBench, the first project-level game code framework dataset and benchmark built on a professional game engine.<n>Our key insight is that Game Jam competitions, community events where developers build complete games under tight time constraints, yield thousands of open-source projects suitable for this purpose.
• Score: 38.658731944117875
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current AI-driven game development has made substantial progress in asset generation, gameplay design, and web-based game coding, yet project-level code engineering on professional game engines remains largely unexplored due to the absence of large-scale datasets and deterministic evaluation methods. We present JamSet and JamBench, the first project-level game code framework dataset and benchmark built on a professional game engine. Our key insight is that Game Jam competitions, community events where developers build complete games under tight time constraints, yield thousands of open-source projects suitable for this purpose. Building on the Godot engine's text-based format and headless execution mode, we design a deterministic verification pipeline from file integrity to runtime behavior collection, distilling 8,133 verified projects from over 240,000 repositories. Of these, 300 manually verified projects form JamBench; the rest constitute JamSet. JamBench defines theme-driven generation and code completion tasks, evaluated through a pipeline combining compilation pass rates, Structural Completeness Score (SCS), and Behavioral Alignment Score (BAS). Evaluation of 9 frontier models reveals a capability cliff as project scale increases, with runtime pass rates dropping from 80.4% on small projects to 5.7% on large ones (Task2a). Code Agents improve compilation rates yet yield no gains in runtime behavioral quality, indicating that the bottleneck lies in architectural design rather than syntactic correctness. Experiments validate JamSet as effective training data. All data and code are publicly available.

Related papers

• GameXpert-Bench: How Far Are Coding Agents from Expert Game Development? [6.806647989372878]
  Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests.<n>Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact.<n>Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent.
  arXiv  Detail & Related papers  (2026-08-22T08:05:16Z)
• GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments [1.392548092257887]
  We present GameEngineBench, a benchmark for evaluating coding agents on scoped C++ implementation tasks inside Unreal Engine 5 projects.<n>The evaluation set consists of 110 tasks spanning gameplay mechanics, multiplayer behavior, AI and world orchestration, animation and movement, UI and session code, loading behavior, online-service integration, persistence, data serialization, XR behavior, and rendering-oriented plugins.
  arXiv  Detail & Related papers  (2026-07-03T17:56:26Z)
• GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine? [65.42976417627254]
  Game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay.<n>We formalize end-to-end game generation as the problem of producing a complete game artifact.<n>We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families.
  arXiv  Detail & Related papers  (2026-06-16T12:34:39Z)
• OpenGame: Open Agentic Coding for Games [18.670337884754577]
  OpenGame is the first open-source agentic framework explicitly designed for end-to-end web game creation.<n>Powering this framework is GameCoder-27B, a code LLM specialized for game engine mastery.<n>OpenGame-Bench is an evaluation pipeline that scores agentic game generation along Build Health, Visual Usability, and Intent Alignment.
  arXiv  Detail & Related papers  (2026-04-20T15:17:03Z)
• Towards Realistic Project-Level Code Generation via Multi-Agent Collaboration and Semantic Architecture Modeling [7.753074942497876]
  We introduce CodeProjectEval, a project-level code generation dataset built from 18 real-world repositories with 12.7 files and 2,388.6 lines of code per task.<n>We propose ProjectGen, a multi-agent framework that decomposes projects into architecture design, skeleton generation, and code filling stages.<n>Experiments show that ProjectGen achieves state-of-the-art performance, passing 52/124 test cases on the small-scale project-level code generation dataset DevBench.
  arXiv  Detail & Related papers  (2025-11-05T12:12:35Z)
• Knowledge Graph-enhanced Large Language Model for Incremental Game PlayTesting [10.112811020571774]
  This paper proposes a KLPEG framework to conduct precise and efficient testing tailored for incremental game updates.<n>The framework constructs and maintains a Knowledge Graph (KG) to systematically model game elements, task dependencies, and causal relationships.<n> Experiments in two representative game environments, Overcooked and Minecraft, demonstrate that KLPEG can more accurately locate functionalities affected by updates.
  arXiv  Detail & Related papers  (2025-11-04T12:40:46Z)
• CodeClash: Benchmarking Goal-Oriented Software Engineering [63.65464283837602]
  We run 1680 tournaments (25,200 rounds total) to evaluate 8 LMs across 6 arenas.<n>Our results reveal that while models exhibit diverse development styles, they share fundamental limitations in strategic reasoning.<n>We open-source CodeClash to advance the study of autonomous, goal-oriented code development.
  arXiv  Detail & Related papers  (2025-11-02T07:42:51Z)
• SwingArena: Competitive Programming Arena for Long-context GitHub Issue Solving [90.32201622392137]
  We present SwingArena, a competitive evaluation framework for Large Language Models (LLMs)<n>Unlike traditional static benchmarks, SwingArena models the collaborative process of software by pairing LLMs as iterations, who generate patches, and reviewers, who create test cases and verify the patches through continuous integration (CI) pipelines.
  arXiv  Detail & Related papers  (2025-05-29T18:28:02Z)
• DevEval: Evaluating Code Generation in Practical Software Projects [52.16841274646796]
  We propose a new benchmark named DevEval, aligned with Developers' experiences in practical projects. DevEval is collected through a rigorous pipeline, containing 2,690 samples from 119 practical projects. We assess five popular LLMs on DevEval and reveal their actual abilities in code generation.
  arXiv  Detail & Related papers  (2024-01-12T06:51:30Z)
• Predicting Defective Visual Code Changes in a Multi-Language AAA Video Game Project [54.20154707138088]
  We focus on constructing visual code defect prediction models that encompass visual code metrics. We test our models using features extracted from the historical agnostic of a AAA video game project. We find that defect prediction models have better performance overall in terms of the area under the ROC curve.
  arXiv  Detail & Related papers  (2023-09-07T00:18:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.