Paint4Poem: A Dataset for Artistic Visualization of Classical Chinese Poems

• URL: http://arxiv.org/abs/2109.11682v1
• Date: Thu, 23 Sep 2021 22:57:16 GMT
• Title: Paint4Poem: A Dataset for Artistic Visualization of Classical Chinese Poems
• Authors: Dan Li, Shuai Wang, Jie Zou, Chang Tian, Elisha Nieuwburg, Fengyuan Sun, Evangelos Kanoulas
• Abstract summary: We construct a new dataset called Paint4Poem. Paint4Poem consists of 301 high-quality poem-painting pairs collected manually from an influential modern Chinese artist. We analyze Paint4Poem regarding poem diversity, painting style, and the semantic relevance between poems and paintings.
• Score: 20.72849584295798
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work we propose a new task: artistic visualization of classical Chinese poems, where the goal is to generatepaintings of a certain artistic style for classical Chinese poems. For this purpose, we construct a new dataset called Paint4Poem. Thefirst part of Paint4Poem consists of 301 high-quality poem-painting pairs collected manually from an influential modern Chinese artistFeng Zikai. As its small scale poses challenges for effectively training poem-to-painting generation models, we introduce the secondpart of Paint4Poem, which consists of 3,648 caption-painting pairs collected manually from Feng Zikai's paintings and 89,204 poem-painting pairs collected automatically from the web. We expect the former to help learning the artist painting style as it containshis most paintings, and the latter to help learning the semantic relevance between poems and paintings. Further, we analyze Paint4Poem regarding poem diversity, painting style, and the semantic relevance between poems and paintings. We create abenchmark for Paint4Poem: we train two representative text-to-image generation models: AttnGAN and MirrorGAN, and evaluate theirperformance regarding painting pictorial quality, painting stylistic relevance, and semantic relevance between poems and paintings.The results indicate that the models are able to generate paintings that have good pictorial quality and mimic Feng Zikai's style, but thereflection of poem semantics is limited. The dataset also poses many interesting research directions on this task, including transferlearning, few-shot learning, text-to-image generation for low-resource data etc. The dataset is publicly available.(https://github.com/paint4poem/paint4poem)

Related papers

• Semi-supervised Chinese Poem-to-Painting Generation via Cycle-consistent Adversarial Networks [2.250406890348191]
  We propose a semi-supervised approach using cycle-consistent adversarial networks to leverage the limited paired data. We introduce novel evaluation metrics to assess the quality, diversity, and consistency of the generated poems and paintings. The proposed model outperforms previous methods, showing promise in capturing the symbolic essence of artistic expression.
  arXiv  Detail & Related papers  (2024-10-25T04:57:44Z)
• Inverse Painting: Reconstructing The Painting Process [24.57538165449989]
  We formulate this as an autoregressive image generation problem, in which an initially blank "canvas" is iteratively updated. The model learns from real artists by training on many painting videos.
  arXiv  Detail & Related papers  (2024-09-30T17:56:52Z)
• DLP-GAN: learning to draw modern Chinese landscape photos with generative adversarial network [20.74857981451259]
  Chinese landscape painting has a unique and artistic style, and its drawing technique is highly abstract in both the use of color and the realistic representation of objects. Previous methods focus on transferring from modern photos to ancient ink paintings, but little attention has been paid to translating landscape paintings into modern photos.
  arXiv  Detail & Related papers  (2024-03-06T04:46:03Z)
• Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region [66.75826549444909]
  Stroke-based rendering aims to recreate an image with a set of strokes. We propose Compositional Neural Painter, which predicts the painting region based on the current canvas. We extend our method to stroke-based style transfer with a novel differentiable distance transform loss.
  arXiv  Detail & Related papers  (2023-09-07T06:27:39Z)
• Reference-based Painterly Inpainting via Diffusion: Crossing the Wild Reference Domain Gap [80.19252970827552]
  RefPaint is a novel task that crosses the wild reference domain gap and implants novel objects into artworks. Our method enables creative painterly image inpainting with reference objects that would otherwise be difficult to achieve.
  arXiv  Detail & Related papers  (2023-07-20T04:51:10Z)
• Text-Guided Synthesis of Eulerian Cinemagraphs [81.20353774053768]
  We introduce Text2Cinemagraph, a fully automated method for creating cinemagraphs from text descriptions. We focus on cinemagraphs of fluid elements, such as flowing rivers, and drifting clouds, which exhibit continuous motion and repetitive textures.
  arXiv  Detail & Related papers  (2023-07-06T17:59:31Z)
• Learning to Generate Poetic Chinese Landscape Painting with Calligraphy [15.33820176664941]
  Polaca is a novel system to generate poetic Chinese landscape painting with calligraphy. It is equipped with three different modules to complete the whole piece of landscape painting artwork.
  arXiv  Detail & Related papers  (2023-05-08T14:10:10Z)
• CCLAP: Controllable Chinese Landscape Painting Generation via Latent Diffusion Model [54.74470985388726]
  controllable Chinese landscape painting generation method named CCLAP. Our method achieves state-of-the-art performance, especially in artfully-composed and artistic conception.
  arXiv  Detail & Related papers  (2023-04-09T04:16:28Z)
• Inversion-Based Style Transfer with Diffusion Models [78.93863016223858]
  Previous arbitrary example-guided artistic image generation methods often fail to control shape changes or convey elements. We propose an inversion-based style transfer method (InST), which can efficiently and accurately learn the key information of an image.
  arXiv  Detail & Related papers  (2022-11-23T18:44:25Z)
• CCPM: A Chinese Classical Poetry Matching Dataset [50.90794811956129]
  We propose a novel task to assess a model's semantic understanding of poetry by poem matching. This task requires the model to select one line of Chinese classical poetry among four candidates according to the modern Chinese translation of a line of poetry. To construct this dataset, we first obtain a set of parallel data of Chinese classical poetry and modern Chinese translation.
  arXiv  Detail & Related papers  (2021-06-03T16:49:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.