Neural Collage Transfer: Artistic Reconstruction via Material Manipulation

• URL: http://arxiv.org/abs/2311.02202v1
• Date: Fri, 3 Nov 2023 19:10:37 GMT
• Title: Neural Collage Transfer: Artistic Reconstruction via Material Manipulation
• Authors: Ganghun Lee, Minji Kim, Yunsu Lee, Minsu Lee, Byoung-Tak Zhang
• Abstract summary: Collage is a creative art form that uses diverse material scraps as a base unit to compose a single image. pixel-wise generation techniques can reproduce a target image in collage style, but it is not a suitable method due to the solid stroke-by-stroke nature of the collage form. We propose a method for learning to make collages via reinforcement learning without the need for demonstrations or collage artwork data.
• Score: 20.72219392904935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Collage is a creative art form that uses diverse material scraps as a base unit to compose a single image. Although pixel-wise generation techniques can reproduce a target image in collage style, it is not a suitable method due to the solid stroke-by-stroke nature of the collage form. While some previous works for stroke-based rendering produced decent sketches and paintings, collages have received much less attention in research despite their popularity as a style. In this paper, we propose a method for learning to make collages via reinforcement learning without the need for demonstrations or collage artwork data. We design the collage Markov Decision Process (MDP), which allows the agent to handle various materials and propose a model-based soft actor-critic to mitigate the agent's training burden derived from the sophisticated dynamics of collage. Moreover, we devise additional techniques such as active material selection and complexity-based multi-scale collage to handle target images at any size and enhance the results' aesthetics by placing relatively more scraps in areas of high complexity. Experimental results show that the trained agent appropriately selected and pasted materials to regenerate the target image into a collage and obtained a higher evaluation score on content and style than pixel-wise generation methods. Code is available at https://github.com/northadventure/CollageRL.

Related papers

• MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation [54.64194935409982]
  We introduce MuLAn: a novel dataset comprising over 44K MUlti-Layer-wise RGBA decompositions. MuLAn is the first photorealistic resource providing instance decomposition and spatial information for high quality images. We aim to encourage the development of novel generation and editing technology, in particular layer-wise solutions.
  arXiv  Detail & Related papers  (2024-04-03T14:58:00Z)
• Image Collage on Arbitrary Shape via Shape-Aware Slicing and Optimization [6.233023267175408]
  We present a shape slicing algorithm and an optimization scheme that can create image collages of arbitrary shapes. Shape-Aware Slicing, which is designed specifically for irregular shapes, takes human perception and shape structure into account to generate visually pleasing partitions.
  arXiv  Detail & Related papers  (2023-11-17T09:41:30Z)
• Composite Diffusion | whole >= \Sigma parts [0.0]
  This paper introduces Composite Diffusion as a means for artists to generate high-quality images by composing from the sub-scenes. We provide a comprehensive and modular method for Composite Diffusion that enables alternative ways of generating, composing, and harmonizing sub-scenes.
  arXiv  Detail & Related papers  (2023-07-25T17:58:43Z)
• CLIP-CLOP: CLIP-Guided Collage and Photomontage [16.460669517251084]
  We design a gradient-based generator to produce collages. It requires the human artist to curate libraries of image patches and to describe (with prompts) the whole image composition. We explore the aesthetic potentials of high-resolution collages, and provide an open-source Google Colab as an artistic tool.
  arXiv  Detail & Related papers  (2022-05-06T11:33:49Z)
• Self-Similarity Priors: Neural Collages as Differentiable Fractal Representations [73.14227103400964]
  We investigate the role of learning in the automated discovery of self-similarity and in its utilization for downstream tasks. We design a novel class of implicit operators, Neural Collages, which represent data as the parameters of a self-referential, structured transformation. We investigate how to leverage the representations produced by Neural Collages in various tasks, including data compression and generation.
  arXiv  Detail & Related papers  (2022-04-15T22:54:23Z)
• Aesthetic Photo Collage with Deep Reinforcement Learning [16.523810962786705]
  Photo collage aims to automatically arrange multiple photos on a given canvas with high aesthetic quality. Deep learning provides a promising way, but owing to the complexity of collage and lack of training data, a solution has yet to be found. We propose a novel pipeline for automatic generation of aspect ratio specified collage and the reinforcement learning technique is introduced in collage for the first time.
  arXiv  Detail & Related papers  (2021-10-19T07:34:48Z)
• Deep Image Compositing [93.75358242750752]
  We propose a new method which can automatically generate high-quality image composites without any user input. Inspired by Laplacian pyramid blending, a dense-connected multi-stream fusion network is proposed to effectively fuse the information from the foreground and background images. Experiments show that the proposed method can automatically generate high-quality composites and outperforms existing methods both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2020-11-04T06:12:24Z)
• Learning to Compose Hypercolumns for Visual Correspondence [57.93635236871264]
  We introduce a novel approach to visual correspondence that dynamically composes effective features by leveraging relevant layers conditioned on the images to match. The proposed method, dubbed Dynamic Hyperpixel Flow, learns to compose hypercolumn features on the fly by selecting a small number of relevant layers from a deep convolutional neural network.
  arXiv  Detail & Related papers  (2020-07-21T04:03:22Z)
• RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval [76.87013602243053]
  We propose a differentiable retrieval module to synthesize images from scene description with retrieved patches as reference. We conduct extensive quantitative and qualitative experiments to demonstrate that the proposed method can generate realistic and diverse images.
  arXiv  Detail & Related papers  (2020-07-16T17:59:04Z)
• SketchyCOCO: Image Generation from Freehand Scene Sketches [71.85577739612579]
  We introduce the first method for automatic image generation from scene-level freehand sketches. Key contribution is an attribute vector bridged Geneversarative Adrial Network called EdgeGAN. We have built a large-scale composite dataset called SketchyCOCO to support and evaluate the solution.
  arXiv  Detail & Related papers  (2020-03-05T14:54:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.