Zero-Shot Text-to-Parameter Translation for Game Character Auto-Creation

• URL: http://arxiv.org/abs/2303.01311v1
• Date: Thu, 2 Mar 2023 14:37:17 GMT
• Title: Zero-Shot Text-to-Parameter Translation for Game Character Auto-Creation
• Authors: Rui Zhao, Wei Li, Zhipeng Hu, Lincheng Li, Zhengxia Zou, Zhenwei Shi, Changjie Fan
• Abstract summary: This paper proposes a novel text-to- parameter translation method (T2P) to achieve zero-shot text-driven game character auto-creation. With our method, users can create a vivid in-game character with arbitrary text description without using any reference photo or editing hundreds of parameters manually.
• Score: 48.62643177644139
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent popular Role-Playing Games (RPGs) saw the great success of character auto-creation systems. The bone-driven face model controlled by continuous parameters (like the position of bones) and discrete parameters (like the hairstyles) makes it possible for users to personalize and customize in-game characters. Previous in-game character auto-creation systems are mostly image-driven, where facial parameters are optimized so that the rendered character looks similar to the reference face photo. This paper proposes a novel text-to-parameter translation method (T2P) to achieve zero-shot text-driven game character auto-creation. With our method, users can create a vivid in-game character with arbitrary text description without using any reference photo or editing hundreds of parameters manually. In our method, taking the power of large-scale pre-trained multi-modal CLIP and neural rendering, T2P searches both continuous facial parameters and discrete facial parameters in a unified framework. Due to the discontinuous parameter representation, previous methods have difficulty in effectively learning discrete facial parameters. T2P, to our best knowledge, is the first method that can handle the optimization of both discrete and continuous parameters. Experimental results show that T2P can generate high-quality and vivid game characters with given text prompts. T2P outperforms other SOTA text-to-3D generation methods on both objective evaluations and subjective evaluations.

Related papers

• GSmoothFace: Generalized Smooth Talking Face Generation via Fine Grained 3D Face Guidance [83.43852715997596]
  GSmoothFace is a novel two-stage generalized talking face generation model guided by a fine-grained 3d face model. It can synthesize smooth lip dynamics while preserving the speaker's identity. Both quantitative and qualitative experiments confirm the superiority of our method in terms of realism, lip synchronization, and visual quality.
  arXiv  Detail & Related papers  (2023-12-12T16:00:55Z)
• ChatAnything: Facetime Chat with LLM-Enhanced Personas [87.76804680223003]
  We propose the mixture of voices (MoV) and the mixture of diffusers (MoD) for diverse voice and appearance generation. For MoV, we utilize the text-to-speech (TTS) algorithms with a variety of pre-defined tones. MoD, we combine the recent popular text-to-image generation techniques and talking head algorithms to streamline the process of generating talking objects.
  arXiv  Detail & Related papers  (2023-11-12T08:29:41Z)
• TADA! Text to Animatable Digital Avatars [57.52707683788961]
  TADA takes textual descriptions and produces expressive 3D avatars with high-quality geometry and lifelike textures. We derive an optimizable high-resolution body model from SMPL-X with 3D displacements and a texture map. We render normals and RGB images of the generated character and exploit their latent embeddings in the SDS training process.
  arXiv  Detail & Related papers  (2023-08-21T17:59:10Z)
• Parametric Implicit Face Representation for Audio-Driven Facial Reenactment [52.33618333954383]
  We propose a novel audio-driven facial reenactment framework that is both controllable and can generate high-quality talking heads. Specifically, our parametric implicit representation parameterizes the implicit representation with interpretable parameters of 3D face models. Our method can generate more realistic results than previous methods with greater fidelity to the identities and talking styles of speakers.
  arXiv  Detail & Related papers  (2023-06-13T07:08:22Z)
• StyleT2F: Generating Human Faces from Textual Description Using StyleGAN2 [0.0]
  StyleT2F is a method of controlling the output of StyleGAN2 using text. Our method proves to capture the required features correctly and shows consistency between the input text and the output images.
  arXiv  Detail & Related papers  (2022-04-17T04:51:30Z)
• MeInGame: Create a Game Character Face from a Single Portrait [15.432712351907012]
  We propose an automatic character face creation method that predicts both facial shape and texture from a single portrait. Experiments show that our method outperforms state-of-the-art methods used in games.
  arXiv  Detail & Related papers  (2021-02-04T02:12:19Z)
• Learning an Animatable Detailed 3D Face Model from In-The-Wild Images [50.09971525995828]
  We present the first approach to jointly learn a model with animatable detail and a detailed 3D face regressor from in-the-wild images. Our DECA model is trained to robustly produce a UV displacement map from a low-dimensional latent representation. We introduce a novel detail-consistency loss to disentangle person-specific details and expression-dependent wrinkles.
  arXiv  Detail & Related papers  (2020-12-07T19:30:45Z)
• S2FGAN: Semantically Aware Interactive Sketch-to-Face Translation [11.724779328025589]
  This paper proposes a sketch-to-image generation framework called S2FGAN. We employ two latent spaces to control the face appearance and adjust the desired attributes of the generated face. Our method successfully outperforms state-of-the-art methods on attribute manipulation by exploiting greater control of attribute intensity.
  arXiv  Detail & Related papers  (2020-11-30T13:42:39Z)
• Neutral Face Game Character Auto-Creation via PokerFace-GAN [0.0]
  This paper studies the problem of automatically creating in-game characters with a single photo. We first build a differentiable character which is more flexible than the previous methods in multi-view rendering cases. We then take advantage of the adversarial training to effectively disentangle the expression parameters from the identity parameters.
  arXiv  Detail & Related papers  (2020-08-17T08:43:48Z)
• Fast and Robust Face-to-Parameter Translation for Game Character Auto-Creation [0.0]
  This paper proposes a game character auto-creation framework that generates in-game characters according to a player's input face photo. Our method shows better robustness than previous methods, especially for those photos with head-pose variance.
  arXiv  Detail & Related papers  (2020-08-17T07:45:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.