CreativeSynth: Creative Blending and Synthesis of Visual Arts based on Multimodal Diffusion

• URL: http://arxiv.org/abs/2401.14066v2
• Date: Tue, 30 Jan 2024 05:58:09 GMT
• Title: CreativeSynth: Creative Blending and Synthesis of Visual Arts based on Multimodal Diffusion
• Authors: Nisha Huang, Weiming Dong, Yuxin Zhang, Fan Tang, Ronghui Li, Chongyang Ma, Xiu Li, Changsheng Xu
• Abstract summary: Large-scale text-to-image generative models have made impressive strides, showcasing their ability to synthesize a vast array of high-quality images. However, adapting these models for artistic image editing presents two significant challenges. We build the innovative unified framework Creative Synth, which is based on a diffusion model with the ability to coordinate multimodal inputs.
• Score: 74.44273919041912
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large-scale text-to-image generative models have made impressive strides, showcasing their ability to synthesize a vast array of high-quality images. However, adapting these models for artistic image editing presents two significant challenges. Firstly, users struggle to craft textual prompts that meticulously detail visual elements of the input image. Secondly, prevalent models, when effecting modifications in specific zones, frequently disrupt the overall artistic style, complicating the attainment of cohesive and aesthetically unified artworks. To surmount these obstacles, we build the innovative unified framework CreativeSynth, which is based on a diffusion model with the ability to coordinate multimodal inputs and multitask in the field of artistic image generation. By integrating multimodal features with customized attention mechanisms, CreativeSynth facilitates the importation of real-world semantic content into the domain of art through inversion and real-time style transfer. This allows for the precise manipulation of image style and content while maintaining the integrity of the original model parameters. Rigorous qualitative and quantitative evaluations underscore that CreativeSynth excels in enhancing artistic images' fidelity and preserves their innate aesthetic essence. By bridging the gap between generative models and artistic finesse, CreativeSynth becomes a custom digital palette.

Related papers

• Neural-Polyptych: Content Controllable Painting Recreation for Diverse Genres [30.83874057768352]
  We present a unified framework, Neural-Polyptych, to facilitate the creation of expansive, high-resolution paintings. We have designed a multi-scale GAN-based architecture to decompose the generation process into two parts. We validate our approach to diverse genres of both Eastern and Western paintings.
  arXiv  Detail & Related papers  (2024-09-29T12:46:00Z)
• ORACLE: Leveraging Mutual Information for Consistent Character Generation with LoRAs in Diffusion Models [3.7599363231894185]
  We introduce a novel framework designed to produce consistent character representations from a single text prompt. Our framework outperforms existing methods in generating characters with consistent visual identities.
  arXiv  Detail & Related papers  (2024-06-04T23:39:08Z)
• FaceStudio: Put Your Face Everywhere in Seconds [23.381791316305332]
  Identity-preserving image synthesis seeks to maintain a subject's identity while adding a personalized, stylistic touch. Traditional methods, such as Textual Inversion and DreamBooth, have made strides in custom image creation. Our research introduces a novel approach to identity-preserving synthesis, with a particular focus on human images.
  arXiv  Detail & Related papers  (2023-12-05T11:02:45Z)
• DreamCreature: Crafting Photorealistic Virtual Creatures from Imagination [140.1641573781066]
  We introduce a novel task, Virtual Creatures Generation: Given a set of unlabeled images of the target concepts, we aim to train a T2I model capable of creating new, hybrid concepts. We propose a new method called DreamCreature, which identifies and extracts the underlying sub-concepts. The T2I thus adapts to generate novel concepts with faithful structures and photorealistic appearance.
  arXiv  Detail & Related papers  (2023-11-27T01:24:31Z)
• Generative AI Model for Artistic Style Transfer Using Convolutional Neural Networks [0.0]
  Artistic style transfer involves fusing the content of one image with the artistic style of another to create unique visual compositions. This paper presents a comprehensive overview of a novel technique for style transfer using Convolutional Neural Networks (CNNs)
  arXiv  Detail & Related papers  (2023-10-27T16:21:17Z)
• Composite Diffusion | whole >= \Sigma parts [0.0]
  This paper introduces Composite Diffusion as a means for artists to generate high-quality images by composing from the sub-scenes. We provide a comprehensive and modular method for Composite Diffusion that enables alternative ways of generating, composing, and harmonizing sub-scenes.
  arXiv  Detail & Related papers  (2023-07-25T17:58:43Z)
• Composer: Creative and Controllable Image Synthesis with Composable Conditions [57.78533372393828]
  Recent large-scale generative models learned on big data are capable of synthesizing incredible images yet suffer from limited controllability. This work offers a new generation paradigm that allows flexible control of the output image, such as spatial layout and palette, while maintaining the synthesis quality and model creativity.
  arXiv  Detail & Related papers  (2023-02-20T05:48:41Z)
• QuantArt: Quantizing Image Style Transfer Towards High Visual Fidelity [94.5479418998225]
  We propose a new style transfer framework called QuantArt for high visual-fidelity stylization. Our framework achieves significantly higher visual fidelity compared with the existing style transfer methods.
  arXiv  Detail & Related papers  (2022-12-20T17:09:53Z)
• DoodleFormer: Creative Sketch Drawing with Transformers [68.18953603715514]
  Creative sketching or doodling is an expressive activity, where imaginative and previously unseen depictions of everyday visual objects are drawn. Here, we propose a novel coarse-to-fine two-stage framework, DoodleFormer, that decomposes the creative sketch generation problem into the creation of coarse sketch composition. To ensure diversity of the generated creative sketches, we introduce a probabilistic coarse sketch decoder.
  arXiv  Detail & Related papers  (2021-12-06T18:59:59Z)
• Modeling Artistic Workflows for Image Generation and Editing [83.43047077223947]
  We propose a generative model that follows a given artistic workflow. It enables both multi-stage image generation as well as multi-stage image editing of an existing piece of art.
  arXiv  Detail & Related papers  (2020-07-14T17:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.