VisioBlend: Sketch and Stroke-Guided Denoising Diffusion Probabilistic Model for Realistic Image Generation

• URL: http://arxiv.org/abs/2407.05209v1
• Date: Wed, 15 May 2024 11:27:27 GMT
• Title: VisioBlend: Sketch and Stroke-Guided Denoising Diffusion Probabilistic Model for Realistic Image Generation
• Authors: Harshkumar Devmurari, Gautham Kuckian, Prajjwal Vishwakarma, Krunali Vartak,
• Abstract summary: We propose a unified framework supporting three-dimensional control over image synthesis from sketches and strokes based on diffusion models. It enables users to decide the level of faithfulness to the input strokes and sketches. It solves the problem of data availability by synthesizing new data points from hand-drawn sketches and strokes.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating images from hand-drawings is a crucial and fundamental task in content creation. The translation is challenging due to the infinite possibilities and the diverse expectations of users. However, traditional methods are often limited by the availability of training data. Therefore, VisioBlend, a unified framework supporting three-dimensional control over image synthesis from sketches and strokes based on diffusion models, is proposed. It enables users to decide the level of faithfulness to the input strokes and sketches. VisioBlend achieves state-of-the-art performance in terms of realism and flexibility, enabling various applications in image synthesis from sketches and strokes. It solves the problem of data availability by synthesizing new data points from hand-drawn sketches and strokes, enriching the dataset and enabling more robust and diverse image synthesis. This work showcases the power of diffusion models in image creation, offering a user-friendly and versatile approach for turning artistic visions into reality.

Related papers

• CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing [21.12815542848095]
  Personalization techniques for large text-to-image (T2I) models allow users to incorporate new concepts from reference images. Existing methods primarily rely on textual descriptions, leading to limited control over customized images. We identify sketches as an intuitive and versatile representation that can facilitate such control.
  arXiv  Detail & Related papers  (2024-02-27T15:52:59Z)
• Sequential Modeling Enables Scalable Learning for Large Vision Models [120.91839619284431]
  We introduce a novel sequential modeling approach which enables learning a Large Vision Model (LVM) without making use of any linguistic data. We define a common format, "visual sentences", in which we can represent raw images and videos as well as annotated data sources.
  arXiv  Detail & Related papers  (2023-12-01T18:59:57Z)
• IT3D: Improved Text-to-3D Generation with Explicit View Synthesis [71.68595192524843]
  This study presents a novel strategy that leverages explicitly synthesized multi-view images to address these issues. Our approach involves the utilization of image-to-image pipelines, empowered by LDMs, to generate posed high-quality images. For the incorporated discriminator, the synthesized multi-view images are considered real data, while the renderings of the optimized 3D models function as fake data.
  arXiv  Detail & Related papers  (2023-08-22T14:39:17Z)
• DiffSketching: Sketch Control Image Synthesis with Diffusion Models [10.172753521953386]
  Deep learning models for sketch-to-image synthesis need to overcome the distorted input sketch without visual details. Our model matches sketches through the cross domain constraints, and uses a classifier to guide the image synthesis more accurately. Our model can beat GAN-based method in terms of generation quality and human evaluation, and does not rely on massive sketch-image datasets.
  arXiv  Detail & Related papers  (2023-05-30T07:59:23Z)
• Reference-based Image Composition with Sketch via Structure-aware Diffusion Model [38.1193912666578]
  We introduce a multi-input-conditioned image composition model that incorporates a sketch as a novel modal, alongside a reference image. Thanks to the edge-level controllability using sketches, our method enables a user to edit or complete an image sub-part. Our framework fine-tunes a pre-trained diffusion model to complete missing regions using the reference image while maintaining sketch guidance.
  arXiv  Detail & Related papers  (2023-03-31T06:12:58Z)
• Person Image Synthesis via Denoising Diffusion Model [116.34633988927429]
  We show how denoising diffusion models can be applied for high-fidelity person image synthesis. Our results on two large-scale benchmarks and a user study demonstrate the photorealism of our proposed approach under challenging scenarios.
  arXiv  Detail & Related papers  (2022-11-22T18:59:50Z)
• Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model [31.652827838300915]
  We propose a unified framework supporting a three-dimensional control over the image synthesis from sketches and strokes based on diffusion models. Our framework achieves state-of-the-art performance while providing flexibility in generating customized images with control over shape, color, and realism. Our method unleashes applications such as editing on real images, generation with partial sketches and strokes, and multi-domain multi-modal synthesis.
  arXiv  Detail & Related papers  (2022-08-26T13:59:26Z)
• DeepFacePencil: Creating Face Images from Freehand Sketches [77.00929179469559]
  Existing image-to-image translation methods require a large-scale dataset of paired sketches and images for supervision. We propose DeepFacePencil, an effective tool that is able to generate photo-realistic face images from hand-drawn sketches.
  arXiv  Detail & Related papers  (2020-08-31T03:35:21Z)
• SketchyCOCO: Image Generation from Freehand Scene Sketches [71.85577739612579]
  We introduce the first method for automatic image generation from scene-level freehand sketches. Key contribution is an attribute vector bridged Geneversarative Adrial Network called EdgeGAN. We have built a large-scale composite dataset called SketchyCOCO to support and evaluate the solution.
  arXiv  Detail & Related papers  (2020-03-05T14:54:10Z)
• Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches [133.01690754567252]
  Sketch-based image editing aims to synthesize and modify photos based on the structural information provided by the human-drawn sketches. Deep Plastic Surgery is a novel, robust and controllable image editing framework that allows users to interactively edit images using hand-drawn sketch inputs.
  arXiv  Detail & Related papers  (2020-01-09T08:57:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.