From Pixels to States: Rethinking Interactive World Models as Game Engines

• URL: http://arxiv.org/abs/2607.14076v1
• Date: Wed, 15 Jul 2026 17:48:48 GMT
• Title: From Pixels to States: Rethinking Interactive World Models as Game Engines
• Abstract summary: This paper examines interactive game world modeling along four dimensions: player action control, game state dynamics, state-observation persistence, and real-time interactive generation.<n>We present a scalable data engine for Black Myth: Wukong that collects over 90 hours of gameplay with frame-aligned player actions, ground-truth game states, and visual observations.
• Score: 32.66997200688335
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Building interactive worlds that respond coherently to player actions has long been a shared goal of computer graphics, games, and artificial intelligence. Recent video generative models provide a data-driven route toward this goal by predicting future observations conditioned on user actions, and are increasingly regarded as potential next-generation game engines. Realizing a genuinely interactive game world, however, requires interaction outcomes that follow rules over evolving game conditions, consequences that persist over long horizons, and a generation loop that operates in real time. Conventional game engines realize these properties through a recurrent action-state-observation loop, in which player actions update an explicit game state according to predefined rules and observations are rendered from the resulting state. Taking this loop as an organizing lens, this paper examines interactive game world modeling along four dimensions: player action control, game state dynamics, state-observation persistence, and real-time interactive generation. For each dimension, we start from the capabilities required by an interactive game world, group existing approaches into representative families, and discuss the strengths and trade-offs of each family. Complementing this analysis, we present a scalable data engine for Black Myth: Wukong that collects over 90 hours of gameplay with frame-aligned player actions, ground-truth game states, and visual observations, together with structured and semantic annotations, as a resource for state-aware game world modeling. We hope this paper offers a clear picture of where the field stands and fosters progress toward interactive game worlds.

Related papers

• Valerant: An Automatic Navigable Game Map Generator via Action-Conditioned World Model Exploration [10.110436795631793]
  World Action Models (WAMs) couple predictive world modeling with action generation, allowing anticipated future states to guide agent behavior.<n>We present textscValerant, a framework that transforms a pretrained action-conditioned world model into a WAM for exploring and constructing 3D game maps.<n>This framework extends WAM-based interaction beyond 2D visual simulation and offers a new approach to reducing manual effort in 3D game-map creation.
  arXiv  Detail & Related papers  (2026-09-08T20:17:48Z)
• GameWAM: A World Action Model for Video Games [36.43342575951464]
  We introduce GameWAM, to our knowledge the first WAM for native closed-loop gameplay and GUI control.<n>To support joint world-action learning, we construct synchronized gameplay and GUI trajectories.<n>Experiments demonstrate competitive task success with fewer executed native actions than the compared agents.
  arXiv  Detail & Related papers  (2026-08-25T17:21:15Z)
• PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives [86.3073067035531]
  Video world models simulate future states conditioned on current observations and user actions.<n>Recent systems have demonstrated impressive video consistency and action controllability over long sequences.<n>But fairly comparing these interactive models remains challenging.<n>We employ multi-modal Agent Players to interact with world models toward specified long-horizon objectives.
  arXiv  Detail & Related papers  (2026-08-13T17:59:30Z)
• StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation [53.07918503798731]
  StatePlay is a novel state-aware game world model that predicts visual content and game states to promote mechanics-consistent generation.<n>Our work highlights the importance of state-aware game world modeling and advances beyond pixel-level realism toward complete and mechanically faithful game generation.
  arXiv  Detail & Related papers  (2026-07-29T10:49:30Z)
• GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine? [65.42976417627254]
  Game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay.<n>We formalize end-to-end game generation as the problem of producing a complete game artifact.<n>We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families.
  arXiv  Detail & Related papers  (2026-06-16T12:34:39Z)
• WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG [38.24292688170687]
  We propose WildWorld, a large-scale action-conditioned world modeling dataset with explicit state annotations.<n>We show challenges in modeling semantically rich actions and maintaining long-horizon state consistency.
  arXiv  Detail & Related papers  (2026-03-24T17:58:25Z)
• MultiGen: Level-Design for Editable Multiplayer Worlds in Diffusion Game Engines [55.641812280108574]
  We introduce an explicit external memory into the system that is continually updated by user actions and queried throughout the generation roll-out.<n>This design gives users direct, editable control over environment structure via an editable memory representation.<n>It naturally extends to real-time multiplayer rollouts with coherent viewpoints and consistent cross-player interactions.
  arXiv  Detail & Related papers  (2026-03-03T18:58:17Z)
• Hunyuan-GameCraft-2: Instruction-following Interactive Game World Model [19.937724706042804]
  Hunyuan-GameCraft-2 is a new paradigm of instruction-driven interaction for generative game world modeling.<n>Our model allows users to control game video contents through natural language prompts, keyboard, or mouse signals.<n>Our model generates temporally coherent and causally grounded interactive game videos.
  arXiv  Detail & Related papers  (2025-11-28T18:26:39Z)
• Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Condition [18.789597877579986]
  Hunyuan-GameCraft is a novel framework for high-dynamic interactive video generation in game environments.<n>To achieve fine-grained action control, we unify standard keyboard and mouse inputs into a shared camera representation space.<n>We propose a hybrid history-conditioned training strategy that extends video sequences autoregressively while preserving game scene information.
  arXiv  Detail & Related papers  (2025-06-20T17:50:37Z)
• MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft [21.530000271719803]
  We propose MineWorld, a real-time interactive world model on Minecraft.<n>MineWorld is driven by a visual-action autoregressive Transformer, which takes paired game scenes and corresponding actions as input.<n>We develop a novel parallel decoding algorithm that predicts the spatial redundant tokens in each frame at the same time.
  arXiv  Detail & Related papers  (2025-04-11T09:41:04Z)
• Unbounded: A Generative Infinite Game of Character Life Simulation [68.37260000219479]
  We introduce the concept of a generative infinite game, a video game that transcends the traditional boundaries of finite, hard-coded systems by using generative models. We leverage recent advances in generative AI to create Unbounded: a game of character life simulation that is fully encapsulated in generative models.
  arXiv  Detail & Related papers  (2024-10-24T17:59:31Z)
• Tachikuma: Understading Complex Interactions with Multi-Character and Novel Objects by Large Language Models [67.20964015591262]
  We introduce a benchmark named Tachikuma, comprising a Multiple character and novel Object based interaction Estimation task and a supporting dataset. The dataset captures log data from real-time communications during gameplay, providing diverse, grounded, and complex interactions for further explorations. We present a simple prompting baseline and evaluate its performance, demonstrating its effectiveness in enhancing interaction understanding.
  arXiv  Detail & Related papers  (2023-07-24T07:40:59Z)
• Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models [68.85478477006178]
  We present a Promptable Game Model (PGM) for neural video game simulators. It allows a user to play the game by prompting it with high- and low-level action sequences. Most captivatingly, our PGM unlocks the director's mode, where the game is played by specifying goals for the agents in the form of a prompt. Our method significantly outperforms existing neural video game simulators in terms of rendering quality and unlocks applications beyond the capabilities of the current state of the art.
  arXiv  Detail & Related papers  (2023-03-23T17:43:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.