AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction

• URL: http://arxiv.org/abs/2504.01014v1
• Date: Tue, 01 Apr 2025 17:57:18 GMT
• Title: AnimeGamer: Infinite Anime Life Simulation with Next Game State Prediction
• Authors: Junhao Cheng, Yuying Ge, Yixiao Ge, Jing Liao, Ying Shan,
• Abstract summary: Recently, a pioneering approach for infinite anime life simulation employs large language models (LLMs) to translate multi-turn text dialogues into language instructions for image generation.<n>We propose AnimeGamer, which is built upon Multimodal Large Language Models (MLLMs) to generate each game state.<n>We introduce novel action-aware multimodal representations to represent animation shots, which can be decoded into high-quality video clips.
• Score: 58.240114139186275
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in image and video synthesis have opened up new promise in generative games. One particularly intriguing application is transforming characters from anime films into interactive, playable entities. This allows players to immerse themselves in the dynamic anime world as their favorite characters for life simulation through language instructions. Such games are defined as infinite game since they eliminate predetermined boundaries and fixed gameplay rules, where players can interact with the game world through open-ended language and experience ever-evolving storylines and environments. Recently, a pioneering approach for infinite anime life simulation employs large language models (LLMs) to translate multi-turn text dialogues into language instructions for image generation. However, it neglects historical visual context, leading to inconsistent gameplay. Furthermore, it only generates static images, failing to incorporate the dynamics necessary for an engaging gaming experience. In this work, we propose AnimeGamer, which is built upon Multimodal Large Language Models (MLLMs) to generate each game state, including dynamic animation shots that depict character movements and updates to character states, as illustrated in Figure 1. We introduce novel action-aware multimodal representations to represent animation shots, which can be decoded into high-quality video clips using a video diffusion model. By taking historical animation shot representations as context and predicting subsequent representations, AnimeGamer can generate games with contextual consistency and satisfactory dynamics. Extensive evaluations using both automated metrics and human evaluations demonstrate that AnimeGamer outperforms existing methods in various aspects of the gaming experience. Codes and checkpoints are available at https://github.com/TencentARC/AnimeGamer.

Related papers

• MoCha: Towards Movie-Grade Talking Character Synthesis [62.007000023747445]
  We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. We propose MoCha, the first of its kind to generate talking characters.
  arXiv  Detail & Related papers  (2025-03-30T04:22:09Z)
• Unbounded: A Generative Infinite Game of Character Life Simulation [68.37260000219479]
  We introduce the concept of a generative infinite game, a video game that transcends the traditional boundaries of finite, hard-coded systems by using generative models. We leverage recent advances in generative AI to create Unbounded: a game of character life simulation that is fully encapsulated in generative models.
  arXiv  Detail & Related papers  (2024-10-24T17:59:31Z)
• GAVEL: Generating Games Via Evolution and Language Models [40.896938709468465]
  We explore the generation of novel games in the Ludii game description language.<n>We train a model that intelligently mutates and recombines games and mechanics expressed as code.<n>A sample of the generated games are available to play online through the Ludii portal.
  arXiv  Detail & Related papers  (2024-07-12T16:08:44Z)
• Promptable Game Models: Text-Guided Game Simulation via Masked Diffusion Models [68.85478477006178]
  We present a Promptable Game Model (PGM) for neural video game simulators. It allows a user to play the game by prompting it with high- and low-level action sequences. Most captivatingly, our PGM unlocks the director's mode, where the game is played by specifying goals for the agents in the form of a prompt. Our method significantly outperforms existing neural video game simulators in terms of rendering quality and unlocks applications beyond the capabilities of the current state of the art.
  arXiv  Detail & Related papers  (2023-03-23T17:43:17Z)
• Language-Guided Face Animation by Recurrent StyleGAN-based Generator [87.56260982475564]
  We study a novel task, language-guided face animation, that aims to animate a static face image with the help of languages. We propose a recurrent motion generator to extract a series of semantic and motion information from the language and feed it along with visual information to a pre-trained StyleGAN to generate high-quality frames.
  arXiv  Detail & Related papers  (2022-08-11T02:57:30Z)
• Triangular Character Animation Sampling with Motion, Emotion, and Relation [78.80083186208712]
  We present a novel framework to sample and synthesize animations by associating the characters' body motions, facial expressions, and social relations. Our method can provide animators with an automatic way to generate 3D character animations, help synthesize interactions between Non-Player Characters (NPCs) and enhance machine emotion intelligence in virtual reality (VR)
  arXiv  Detail & Related papers  (2022-03-09T18:19:03Z)
• Keep CALM and Explore: Language Models for Action Generation in Text-based Games [27.00685301984832]
  We propose the Contextual Action Language Model (CALM) to generate a compact set of action candidates at each game state. We combine CALM with a reinforcement learning agent which re-ranks the generated action candidates to maximize in-game rewards.
  arXiv  Detail & Related papers  (2020-10-06T17:36:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.