Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training

• URL: http://arxiv.org/abs/2608.24680v1
• Date: Tue, 25 Aug 2026 15:13:32 GMT
• Title: Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training
• Abstract summary: We introduce GameUI-Taxonomy and G2WEngine, a full-stack framework that formalizes gameplay UI grounding and removal.<n>G2WEngine automatically extracts reusable UI assets from real gameplay videos and synthesizes temporally coherent UI overlays on clean footage.<n>Based on Game2World, we propose GameCleaner, a mask-free gameplay UI removal model that combines multimodal semantic understanding with video editing capabilities.
• Score: 10.094308570527923
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video games provide a scalable source of training data for video world models, offering diverse environments, complex interactions, and abundant in-the-wild gameplay videos. However, raw gameplay footage entangles the game world with screen-space interfaces, introducing game-specific biases and irrelevant dynamics that hinder world-model training. To address this problem, we introduce GameUI-Taxonomy and G2WEngine, a full-stack framework that formalizes gameplay UI grounding and removal. G2WEngine automatically extracts reusable UI assets from real gameplay videos and synthesizes temporally coherent UI overlays on clean footage. Using this engine, we construct Game2World, comprising 96K synthetic paired videos with precise reconstruction targets and 1,079 in-the-wild clips from 303 games for realistic evaluation. Its asset library contains 5,132 verified UI elements across 21 taxonomy categories, collected from 1,010 representative gameplay frames. Based on Game2World, we propose GameCleaner, a mask-free gameplay UI removal model that combines multimodal semantic understanding with video editing capabilities. Unlike mask-based methods, GameCleaner directly identifies and removes diverse HUD elements while preserving the underlying scene content and temporal dynamics. In a controlled pilot, world models trained on UI-free gameplay improve overall VideoReward by 6.83% over those trained on UI-overlaid data. On UI-removal evaluation, GameCleaner achieves an average AAR of 95.36 on synthetic videos, outperforming the strongest temporal mask baseline by 57.3%, and obtains the best in-the-wild AAR of 80.05 with 99.8 background preservation. These results demonstrate the scalable potential of transforming Internet gameplay videos into high-quality world-model training data. Code, dataset, and model will be available at https://github.com/Dongping-Chen/Game2World.

Related papers

• EgoCS-400K: An Egocentric Gameplay Dataset for World Models [53.9389499966106]
  We introduce EgoCS-400K, a large-scale replay-grounded egocentric Counter-Strike dataset for world models.<n>It is built from public professional CS and CS2 match demos that preserve human gameplay trajectories.<n>EgoCS-400K contains over 400,000 first-person videos and 10,000 hours of gameplay from more than 1,000 matches and 40,000 rounds.
  arXiv  Detail & Related papers  (2026-06-16T17:13:58Z)
• GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine? [65.42976417627254]
  Game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay.<n>We formalize end-to-end game generation as the problem of producing a complete game artifact.<n>We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families.
  arXiv  Detail & Related papers  (2026-06-16T12:34:39Z)
• Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos [82.4003989236851]
  We propose a novel paradigm that leverages glitches in gameplay videos, referring to visual anomalies that violate predefined physical laws, as a rich and scalable supervision source for physical world understanding.<n>We introduce PhysGame, a dataset containing 140,057 glitch-centric question-answer pairs across five physical domains and sixteen fine-grained categories.<n>Experiments show that PhysGame significantly enhances both Game2Real transferability, improving the real world physical reasoning performance of Qwen2.5VL by 2.5%, and Game2General transferability, yielding a 1.9% gain on the MVBench benchmark.
  arXiv  Detail & Related papers  (2026-01-23T06:02:07Z)
• NitroGen: An Open Foundation Model for Generalist Gaming Agents [101.41866522979548]
  NitroGen is a vision-action foundation model for generalist gaming agents.<n>It is trained on 40,000 hours of gameplay videos across more than 1,000 games.
  arXiv  Detail & Related papers  (2026-01-04T16:24:50Z)
• Automated Bug Frame Retrieval from Gameplay Videos Using Vision-Language Models [47.63488459021783]
  We introduce a pipeline that reduces each video to a single frame that best matches the reported bug description.<n>Our approach dramatically reduces manual effort and speeds up triage and regression checks.<n>It offers practical benefits to quality assurance teams and developers across the game industry.
  arXiv  Detail & Related papers  (2025-08-06T21:52:15Z)
• VideoGameBench: Can Vision-Language Models complete popular video games? [8.5302862604852]
  Video games are crafted to be intuitive for humans to learn and master by leveraging innate inductive biases.<n>We introduce VideoGameBench, a benchmark consisting of 10 popular video games from the 1990s that VLMs directly interact with in real-time.<n>We show that frontier vision-language models struggle to progress beyond the beginning of each game.
  arXiv  Detail & Related papers  (2025-05-23T17:43:27Z)
• Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models [68.85478477006178]
  We present a Promptable Game Model (PGM) for neural video game simulators. It allows a user to play the game by prompting it with high- and low-level action sequences. Most captivatingly, our PGM unlocks the director's mode, where the game is played by specifying goals for the agents in the form of a prompt. Our method significantly outperforms existing neural video game simulators in terms of rendering quality and unlocks applications beyond the capabilities of the current state of the art.
  arXiv  Detail & Related papers  (2023-03-23T17:43:17Z)
• CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning [4.168157981135698]
  We propose a search method that accepts any English text query as input to retrieve relevant gameplay videos. Our approach does not rely on any external information (such as video metadata) An example application of our approach is as a gameplay video search engine to aid in reproducing video game bugs.
  arXiv  Detail & Related papers  (2022-03-21T16:23:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.