Looks Like Magic: Transfer Learning in GANs to Generate New Card Illustrations

• URL: http://arxiv.org/abs/2205.14442v1
• Date: Sat, 28 May 2022 14:02:09 GMT
• Title: Looks Like Magic: Transfer Learning in GANs to Generate New Card Illustrations
• Authors: Matheus K. Venturelli, Pedro H. Gomes, J\^onatas Wehrmann
• Abstract summary: We introduce a novel dataset, named MTG, with thousands of illustration from diverse card types and rich in metadata. We show that simpler models, such as DCGANs, are not able to learn to generate proper illustrations in any setting. We perform experiments to understand how well pre-trained features from StyleGan2 can be transferred towards the target domain.
• Score: 5.006086647446482
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, we propose MAGICSTYLEGAN and MAGICSTYLEGAN-ADA - both incarnations of the state-of-the-art models StyleGan2 and StyleGan2 ADA - to experiment with their capacity of transfer learning into a rather different domain: creating new illustrations for the vast universe of the game "Magic: The Gathering" cards. This is a challenging task especially due to the variety of elements present in these illustrations, such as humans, creatures, artifacts, and landscapes - not to mention the plethora of art styles of the images made by various artists throughout the years. To solve the task at hand, we introduced a novel dataset, named MTG, with thousands of illustration from diverse card types and rich in metadata. The resulting set is a dataset composed by a myriad of both realistic and fantasy-like illustrations. Although, to investigate effects of diversity we also introduced subsets that contain specific types of concepts, such as forests, islands, faces, and humans. We show that simpler models, such as DCGANs, are not able to learn to generate proper illustrations in any setting. On the other side, we train instances of MAGICSTYLEGAN using all proposed subsets, being able to generate high quality illustrations. We perform experiments to understand how well pre-trained features from StyleGan2 can be transferred towards the target domain. We show that in well trained models we can find particular instances of noise vector that realistically represent real images from the dataset. Moreover, we provide both quantitative and qualitative studies to support our claims, and that demonstrate that MAGICSTYLEGAN is the state-of-the-art approach for generating Magic illustrations. Finally, this paper highlights some emerging properties regarding transfer learning in GANs, which is still a somehow under-explored field in generative learning research.

Related papers

• HaSPeR: An Image Repository for Hand Shadow Puppet Recognition [2.048226951354646]
  Hand shadow puppetry, also known as shadowgraphy or ombromanie, is a form of theatrical art and storytelling. We present a novel dataset consisting of 8,340 images of hand shadow puppets across 11 classes extracted from both professional and amateur hand shadow puppeteer clips. Our findings show a substantial performance superiority of traditional convolutional models over attention-based transformer architectures.
  arXiv  Detail & Related papers  (2024-08-19T18:56:24Z)
• DreamCreature: Crafting Photorealistic Virtual Creatures from Imagination [140.1641573781066]
  We introduce a novel task, Virtual Creatures Generation: Given a set of unlabeled images of the target concepts, we aim to train a T2I model capable of creating new, hybrid concepts. We propose a new method called DreamCreature, which identifies and extracts the underlying sub-concepts. The T2I thus adapts to generate novel concepts with faithful structures and photorealistic appearance.
  arXiv  Detail & Related papers  (2023-11-27T01:24:31Z)
• Diffusion Based Augmentation for Captioning and Retrieval in Cultural Heritage [28.301944852273746]
  This paper introduces a novel approach to address the challenges of limited annotated data and domain shifts in the cultural heritage domain. By leveraging generative vision-language models, we augment art datasets by generating diverse variations of artworks conditioned on their captions.
  arXiv  Detail & Related papers  (2023-08-14T13:59:04Z)
• Semantically-aware Mask CycleGAN for Translating Artistic Portraits to Photo-realistic Visualizations [0.0]
  I propose the Semantic-aware Mask CycleGAN architecture which can translate artistic portraits to visualizations. This model can generate realistic human portraits by feeding the discriminators semantically masked fake samples. Experiments have shown that the SMCycleGAN generate images with significantly increased realism and minimal loss of content representations.
  arXiv  Detail & Related papers  (2023-06-11T03:58:09Z)
• A domain adaptive deep learning solution for scanpath prediction of paintings [66.46953851227454]
  This paper focuses on the eye-movement analysis of viewers during the visual experience of a certain number of paintings. We introduce a new approach to predicting human visual attention, which impacts several cognitive functions for humans. The proposed new architecture ingests images and returns scanpaths, a sequence of points featuring a high likelihood of catching viewers' attention.
  arXiv  Detail & Related papers  (2022-09-22T22:27:08Z)
• InvGAN: Invertible GANs [88.58338626299837]
  InvGAN, short for Invertible GAN, successfully embeds real images to the latent space of a high quality generative model. This allows us to perform image inpainting, merging, and online data augmentation.
  arXiv  Detail & Related papers  (2021-12-08T21:39:00Z)
• MISS GAN: A Multi-IlluStrator Style Generative Adversarial Network for image to illustration translation [39.57350884615545]
  Multi-IlluStrator Style Generative Adversarial Network (MISS GAN) is a framework for unsupervised image-to-illustration translation. MISS GAN is both input image specific and uses the information of other images using only one trained model.
  arXiv  Detail & Related papers  (2021-08-12T12:23:28Z)
• Level generation and style enhancement -- deep learning for game development overview [0.0]
  We present seven approaches to create level maps, each using statistical methods, machine learning, or deep learning. We aim to present new possibilities for game developers and level artists.
  arXiv  Detail & Related papers  (2021-07-15T15:24:43Z)
• Factors of Influence for Transfer Learning across Diverse Appearance Domains and Task Types [50.1843146606122]
  A simple form of transfer learning is common in current state-of-the-art computer vision models. Previous systematic studies of transfer learning have been limited and the circumstances in which it is expected to work are not fully understood. In this paper we carry out an extensive experimental exploration of transfer learning across vastly different image domains.
  arXiv  Detail & Related papers  (2021-03-24T16:24:20Z)
• Unsupervised Discovery of Disentangled Manifolds in GANs [74.24771216154105]
  Interpretable generation process is beneficial to various image editing applications. We propose a framework to discover interpretable directions in the latent space given arbitrary pre-trained generative adversarial networks.
  arXiv  Detail & Related papers  (2020-11-24T02:18:08Z)
• SketchEmbedNet: Learning Novel Concepts by Imitating Drawings [125.45799722437478]
  We explore properties of image representations learned by training a model to produce sketches of images. We show that this generative, class-agnostic model produces informative embeddings of images from novel examples, classes, and even novel datasets in a few-shot setting.
  arXiv  Detail & Related papers  (2020-08-27T16:43:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.