GameFactory: Creating New Games with Generative Interactive Videos

• URL: http://arxiv.org/abs/2501.08325v2
• Date: Tue, 25 Mar 2025 03:34:45 GMT
• Title: GameFactory: Creating New Games with Generative Interactive Videos
• Authors: Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, Xihui Liu,
• Abstract summary: Generative videos have the potential to revolutionize game development by autonomously creating new content.<n>We present GameFactory, a framework for action-controlled scene-generalizable game video generation.<n> Experimental results demonstrate that GameFactory effectively generates open-domain action-controllable game videos.
• Score: 32.98135338530966
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative videos have the potential to revolutionize game development by autonomously creating new content. In this paper, we present GameFactory, a framework for action-controlled scene-generalizable game video generation. We first address the fundamental challenge of action controllability by introducing GF-Minecraft, a action-annotated game video dataset without human bias, and developing a action control module that enables precise control over both keyboard and mouse inputs. We further extend to support autoregressive generation for unlimited-length interactive videos. More importantly, GameFactory tackles the critical challenge of scene-generalizable action control, which most existing methods fail to address. To enable the creation of entirely new and diverse games beyond fixed styles and scenes, we leverage the open-domain generative priors from pre-trained video diffusion models. To bridge the domain gap between open-domain priors and small-scale game datasets, we propose a multi-phase training strategy with a domain adapter that decouples game style learning from action control. This decoupling ensures that action control learning is no longer bound to specific game styles, thereby achieving scene-generalizable action control. Experimental results demonstrate that GameFactory effectively generates open-domain action-controllable game videos, representing a significant step forward in AI-driven game generation. Our dataset and project page are publicly available at https://yujiwen.github.io/gamefactory/.

Related papers

• Cultivating Game Sense for Yourself: Making VLMs Gaming Experts [23.370716496046217]
  We propose a paradigm shift in gameplay agent design. Instead of directly controlling gameplay, VLM develops specialized execution modules tailored for tasks like shooting and combat. These modules handle real-time game interactions, elevating VLM to a high-level developer.
  arXiv  Detail & Related papers  (2025-03-27T08:40:47Z)
• ReCamMaster: Camera-Controlled Generative Rendering from A Single Video [72.42376733537925]
  ReCamMaster is a camera-controlled generative video re-rendering framework. It reproduces the dynamic scene of an input video at novel camera trajectories. Our method also finds promising applications in video stabilization, super-resolution, and outpainting.
  arXiv  Detail & Related papers  (2025-03-14T17:59:31Z)
• GameGen-X: Interactive Open-world Game Video Generation [10.001128258269675]
  We introduce GameGen-X, the first diffusion transformer model specifically designed for both generating and interactively controlling open-world game videos.<n>It simulates an array of game engine features, such as innovative characters, dynamic environments, complex actions, and diverse events.<n>It provides interactive controllability, predicting and future altering content based on the current clip, thus allowing for gameplay simulation.
  arXiv  Detail & Related papers  (2024-11-01T17:59:17Z)
• Instruction-Driven Game Engines on Large Language Models [59.280666591243154]
  The IDGE project aims to democratize game development by enabling a large language model to follow free-form game rules. We train the IDGE in a curriculum manner that progressively increases the model's exposure to complex scenarios. Our initial progress lies in developing an IDGE for Poker, a universally cherished card game.
  arXiv  Detail & Related papers  (2024-03-30T08:02:16Z)
• Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models [68.85478477006178]
  We present a Promptable Game Model (PGM) for neural video game simulators. It allows a user to play the game by prompting it with high- and low-level action sequences. Most captivatingly, our PGM unlocks the director's mode, where the game is played by specifying goals for the agents in the form of a prompt. Our method significantly outperforms existing neural video game simulators in terms of rendering quality and unlocks applications beyond the capabilities of the current state of the art.
  arXiv  Detail & Related papers  (2023-03-23T17:43:17Z)
• Multi-Game Decision Transformers [49.257185338595434]
  We show that a single transformer-based model can play a suite of up to 46 Atari games simultaneously at close-to-human performance. We compare several approaches in this multi-game setting, such as online and offline RL methods and behavioral cloning. We find that our Multi-Game Decision Transformer models offer the best scalability and performance.
  arXiv  Detail & Related papers  (2022-05-30T16:55:38Z)
• Playable Video Generation [47.531594626822155]
  We aim at allowing a user to control the generated video by selecting a discrete action at every time step as when playing a video game. The difficulty of the task lies both in learning semantically consistent actions and in generating realistic videos conditioned on the user input. We propose a novel framework for PVG that is trained in a self-supervised manner on a large dataset of unlabelled videos.
  arXiv  Detail & Related papers  (2021-01-28T18:55:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.