Interactive Neural Painting

• URL: http://arxiv.org/abs/2307.16441v1
• Date: Mon, 31 Jul 2023 07:02:00 GMT
• Title: Interactive Neural Painting
• Authors: Elia Peruzzo, Willi Menapace, Vidit Goel, Federica Arrigoni, Hao Tang, Xingqian Xu, Arman Chopikyan, Nikita Orlov, Yuxiao Hu, Humphrey Shi, Nicu Sebe, Elisa Ricci
• Abstract summary: This paper proposes the first approach for Interactive Neural Painting (NP) We propose I-Paint, a novel method based on a conditional transformer Variational AutoEncoder (VAE) architecture with a two-stage decoder. Our experiments show that our approach provides good stroke suggestions and compares favorably to the state of the art.
• Score: 66.9376011879115
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In the last few years, Neural Painting (NP) techniques became capable of producing extremely realistic artworks. This paper advances the state of the art in this emerging research domain by proposing the first approach for Interactive NP. Considering a setting where a user looks at a scene and tries to reproduce it on a painting, our objective is to develop a computational framework to assist the users creativity by suggesting the next strokes to paint, that can be possibly used to complete the artwork. To accomplish such a task, we propose I-Paint, a novel method based on a conditional transformer Variational AutoEncoder (VAE) architecture with a two-stage decoder. To evaluate the proposed approach and stimulate research in this area, we also introduce two novel datasets. Our experiments show that our approach provides good stroke suggestions and compares favorably to the state of the art. Additional details, code and examples are available at https://helia95.github.io/inp-website.

Related papers

• AttentionPainter: An Efficient and Adaptive Stroke Predictor for Scene Painting [82.54770866332456]
  Stroke-based Rendering (SBR) aims to decompose an input image into a sequence of parameterized strokes, which can be rendered into a painting that resembles the input image. We propose AttentionPainter, an efficient and adaptive model for single-step neural painting.
  arXiv  Detail & Related papers  (2024-10-21T18:36:45Z)
• Sketch-guided Image Inpainting with Partial Discrete Diffusion Process [5.005162730122933]
  We introduce a novel partial discrete diffusion process (PDDP) for sketch-guided inpainting. PDDP corrupts the masked regions of the image and reconstructs these masked regions conditioned on hand-drawn sketches. The proposed novel transformer module accepts two inputs -- the image containing the masked region to be inpainted and the query sketch to model the reverse diffusion process.
  arXiv  Detail & Related papers  (2024-04-18T07:07:38Z)
• HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models [59.01600111737628]
  HD-Painter is a training free approach that accurately follows prompts and coherently scales to high resolution image inpainting. To this end, we design the Prompt-Aware Introverted Attention (PAIntA) layer enhancing self-attention scores. Our experiments demonstrate that HD-Painter surpasses existing state-of-the-art approaches quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-12-21T18:09:30Z)
• Collaborative Neural Painting [27.880814775833578]
  We introduce a novel task, Collaborative Neural Painting (CNP), to facilitate collaborative art painting generation between humans and machines. CNP should produce a sequence of strokes supporting the completion of a coherent painting. We propose a painting representation based on a sequence of parametrized strokes, which makes it easy both editing and composition operations.
  arXiv  Detail & Related papers  (2023-12-04T10:45:12Z)
• Reconstructing Existing Levels through Level Inpainting [3.1788052710897707]
  This paper introduces Content Augmentation and focuses on the subproblem of level inpainting. We present two approaches for level inpainting: an Autoencoder and a U-net.
  arXiv  Detail & Related papers  (2023-09-18T04:10:27Z)
• Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region [66.75826549444909]
  Stroke-based rendering aims to recreate an image with a set of strokes. We propose Compositional Neural Painter, which predicts the painting region based on the current canvas. We extend our method to stroke-based style transfer with a novel differentiable distance transform loss.
  arXiv  Detail & Related papers  (2023-09-07T06:27:39Z)
• Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing [23.143394242978125]
  paint2pix learns to predict "what a user wants to draw" from rudimentary brushstroke inputs. paint2pix can be used for progressive image synthesis from scratch. Our approach also forms a surprisingly convenient approach for real image editing.
  arXiv  Detail & Related papers  (2022-08-17T06:08:11Z)
• Sketch-Guided Scenery Image Outpainting [83.6612152173028]
  We propose an encoder-decoder based network to conduct sketch-guided outpainting. We apply a holistic alignment module to make the synthesized part be similar to the real one from the global view. Second, we reversely produce the sketches from the synthesized part and encourage them be consistent with the ground-truth ones.
  arXiv  Detail & Related papers  (2020-06-17T11:34:36Z)
• Very Long Natural Scenery Image Prediction by Outpainting [96.8509015981031]
  Outpainting receives less attention due to two challenges in it. First challenge is how to keep the spatial and content consistency between generated images and original input. Second challenge is how to maintain high quality in generated results.
  arXiv  Detail & Related papers  (2019-12-29T16:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.