ProcessPainter: Learn Painting Process from Sequence Data

• URL: http://arxiv.org/abs/2406.06062v2
• Date: Sat, 20 Jul 2024 07:23:19 GMT
• Title: ProcessPainter: Learn Painting Process from Sequence Data
• Authors: Yiren Song, Shijie Huang, Chen Yao, Xiaojun Ye, Hai Ci, Jiaming Liu, Yuxuan Zhang, Mike Zheng Shou,
• Abstract summary: The painting process of artists is inherently stepwise and varies significantly among different painters and styles. Traditional stroke-based rendering methods break down images into sequences of brushstrokes, yet they fall short of replicating the authentic processes of artists. We introduce ProcessPainter, a text-to-video model that is initially pre-trained on synthetic data and subsequently fine-tuned with a select set of artists' painting sequences.
• Score: 27.9875429986135
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The painting process of artists is inherently stepwise and varies significantly among different painters and styles. Generating detailed, step-by-step painting processes is essential for art education and research, yet remains largely underexplored. Traditional stroke-based rendering methods break down images into sequences of brushstrokes, yet they fall short of replicating the authentic processes of artists, with limitations confined to basic brushstroke modifications. Text-to-image models utilizing diffusion processes generate images through iterative denoising, also diverge substantially from artists' painting process. To address these challenges, we introduce ProcessPainter, a text-to-video model that is initially pre-trained on synthetic data and subsequently fine-tuned with a select set of artists' painting sequences using the LoRA model. This approach successfully generates painting processes from text prompts for the first time. Furthermore, we introduce an Artwork Replication Network capable of accepting arbitrary-frame input, which facilitates the controlled generation of painting processes, decomposing images into painting sequences, and completing semi-finished artworks. This paper offers new perspectives and tools for advancing art education and image generation technology.

Related papers

• Inverse Painting: Reconstructing The Painting Process [24.57538165449989]
  We formulate this as an autoregressive image generation problem, in which an initially blank "canvas" is iteratively updated. The model learns from real artists by training on many painting videos.
  arXiv  Detail & Related papers  (2024-09-30T17:56:52Z)
• Learning Inclusion Matching for Animation Paint Bucket Colorization [76.4507878427755]
  We introduce a new learning-based inclusion matching pipeline, which directs the network to comprehend the inclusion relationships between segments. Our method features a two-stage pipeline that integrates a coarse color warping module with an inclusion matching module. To facilitate the training of our network, we also develope a unique dataset, referred to as PaintBucket-Character.
  arXiv  Detail & Related papers  (2024-03-27T08:32:48Z)
• Fill in the ____ (a Diffusion-based Image Inpainting Pipeline) [0.0]
  Inpainting is the process of taking an image and generating lost or intentionally occluded portions. Modern inpainting techniques have shown remarkable ability in generating sensible completions. A critical gap in these existing models will be addressed, focusing on the ability to prompt and control what exactly is generated.
  arXiv  Detail & Related papers  (2024-03-24T05:26:55Z)
• HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models [59.01600111737628]
  HD-Painter is a training free approach that accurately follows prompts and coherently scales to high resolution image inpainting. To this end, we design the Prompt-Aware Introverted Attention (PAIntA) layer enhancing self-attention scores. Our experiments demonstrate that HD-Painter surpasses existing state-of-the-art approaches quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-12-21T18:09:30Z)
• Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region [66.75826549444909]
  Stroke-based rendering aims to recreate an image with a set of strokes. We propose Compositional Neural Painter, which predicts the painting region based on the current canvas. We extend our method to stroke-based style transfer with a novel differentiable distance transform loss.
  arXiv  Detail & Related papers  (2023-09-07T06:27:39Z)
• PaintSeg: Training-free Segmentation via Painting [50.17936803209125]
  PaintSeg is a new unsupervised method for segmenting objects without any training. Inpainting and outpainting are alternated, with the former masking the foreground and filling in the background, and the latter masking the background while recovering the missing part of the foreground object. Our experimental results demonstrate that PaintSeg outperforms existing approaches in coarse mask-prompt, box-prompt, and point-prompt segmentation tasks.
  arXiv  Detail & Related papers  (2023-05-30T20:43:42Z)
• Inversion-Based Style Transfer with Diffusion Models [78.93863016223858]
  Previous arbitrary example-guided artistic image generation methods often fail to control shape changes or convey elements. We propose an inversion-based style transfer method (InST), which can efficiently and accurately learn the key information of an image.
  arXiv  Detail & Related papers  (2022-11-23T18:44:25Z)
• Perceptual Artifacts Localization for Inpainting [60.5659086595901]
  We propose a new learning task of automatic segmentation of inpainting perceptual artifacts. We train advanced segmentation networks on a dataset to reliably localize inpainting artifacts within inpainted images. We also propose a new evaluation metric called Perceptual Artifact Ratio (PAR), which is the ratio of objectionable inpainted regions to the entire inpainted area.
  arXiv  Detail & Related papers  (2022-08-05T18:50:51Z)
• Toward Modeling Creative Processes for Algorithmic Painting [12.602935529346063]
  The paper argues that creative processes often involve two important components: vague, high-level goals and exploratory processes for discovering new ideas. This paper sketches out possible computational mechanisms for imitating those elements of the painting process, including underspecified loss functions and iterative painting procedures with explicit task decompositions.
  arXiv  Detail & Related papers  (2022-05-03T16:33:45Z)
• Intelli-Paint: Towards Developing Human-like Painting Agents [19.261822105543175]
  We propose a novel painting approach which learns to generate output canvases while exhibiting a more human-like painting style. Intelli-Paint consists of 1) a progressive layering strategy which allows the agent to first paint a natural background scene representation before adding in each of the foreground objects in a progressive fashion. We also introduce a novel sequential brushstroke guidance strategy which helps the painting agent to shift its attention between different image regions in a semantic-aware manner.
  arXiv  Detail & Related papers  (2021-12-16T14:56:32Z)
• Generative Art Using Neural Visual Grammars and Dual Encoders [25.100664361601112]
  A novel algorithm for producing generative art is described. It allows a user to input a text string, and which in a creative response to this string, outputs an image.
  arXiv  Detail & Related papers  (2021-05-01T04:21:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.