VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models

• URL: http://arxiv.org/abs/2211.11319v1
• Date: Mon, 21 Nov 2022 10:04:27 GMT
• Title: VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models
• Authors: Ajay Jain and Amber Xie and Pieter Abbeel
• Abstract summary: We show that a text-conditioned diffusion model trained on pixel representations of images can be used to generate SVG-exportable vector graphics. Inspired by recent text-to-3D work, we learn an SVG consistent with a caption using Score Distillation Sampling. Experiments show greater quality than prior work, and demonstrate a range of styles including pixel art and sketches.
• Score: 82.93345261434943
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diffusion models have shown impressive results in text-to-image synthesis. Using massive datasets of captioned images, diffusion models learn to generate raster images of highly diverse objects and scenes. However, designers frequently use vector representations of images like Scalable Vector Graphics (SVGs) for digital icons or art. Vector graphics can be scaled to any size, and are compact. We show that a text-conditioned diffusion model trained on pixel representations of images can be used to generate SVG-exportable vector graphics. We do so without access to large datasets of captioned SVGs. By optimizing a differentiable vector graphics rasterizer, our method, VectorFusion, distills abstract semantic knowledge out of a pretrained diffusion model. Inspired by recent text-to-3D work, we learn an SVG consistent with a caption using Score Distillation Sampling. To accelerate generation and improve fidelity, VectorFusion also initializes from an image sample. Experiments show greater quality than prior work, and demonstrate a range of styles including pixel art and sketches. See our project webpage at https://ajayj.com/vectorfusion .

Related papers

• Vector Grimoire: Codebook-based Shape Generation under Raster Image Supervision [20.325246638505714]
  We introduce GRIMOIRE, a text-guided generative model that learns to map images onto a discrete codebook by reconstructing them as vector shapes. Unlike existing models that require direct supervision from data, GRIMOIRE learns using only image supervision which opens up vector generative modeling to significantly more data.
  arXiv  Detail & Related papers  (2024-10-08T12:41:31Z)
• SuperSVG: Superpixel-based Scalable Vector Graphics Synthesis [66.44553285020066]
  SuperSVG is a superpixel-based vectorization model that achieves fast and high-precision image vectorization. We propose a two-stage self-training framework, where a coarse-stage model is employed to reconstruct the main structure and a refinement-stage model is used for enriching the details. Experiments demonstrate the superior performance of our method in terms of reconstruction accuracy and inference time compared to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-06-14T07:43:23Z)
• StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis [112.25071764647683]
  StrokeNUWA is a pioneering work exploring a better visual representation ''stroke tokens'' on vector graphics. equipped with stroke tokens, StrokeNUWA can significantly surpass traditional LLM-based and optimization-based methods. StrokeNUWA achieves up to a 94x speedup in inference over the speed of prior methods with an exceptional SVG code compression ratio of 6.9%.
  arXiv  Detail & Related papers  (2024-01-30T15:20:26Z)
• SVGDreamer: Text Guided SVG Generation with Diffusion Model [31.76771064173087]
  We propose a novel text-guided vector graphics synthesis method called SVGDreamer. SIVE process enables decomposition of synthesis into foreground objects and background. VPSD approach addresses issues of shape over-smoothing, color over-saturation, limited diversity, and slow convergence.
  arXiv  Detail & Related papers  (2023-12-27T08:50:01Z)
• SAMVG: A Multi-stage Image Vectorization Model with the Segment-Anything Model [59.40189857428461]
  We propose a multi-stage model to vectorize images into SVG (Scalable Vector Graphics) Firstly, SAMVG uses general image segmentation provided by the Segment-Anything Model and uses a novel filtering method to identify the best dense segmentation map for the entire image. Secondly, SAMVG then identifies missing components and adds more detailed components to the SVG.
  arXiv  Detail & Related papers  (2023-11-09T11:11:56Z)
• Text-Guided Vector Graphics Customization [31.41266632288932]
  We propose a novel pipeline that generates high-quality customized vector graphics based on textual prompts. Our method harnesses the capabilities of large pre-trained text-to-image models. We evaluate our method using multiple metrics from vector-level, image-level and text-level perspectives.
  arXiv  Detail & Related papers  (2023-09-21T17:59:01Z)
• Towards Layer-wise Image Vectorization [57.26058135389497]
  We propose Layerwise Image Vectorization, namely LIVE, to convert images to SVGs and simultaneously maintain its image topology. Live generates compact forms with layer-wise structures that are semantically consistent with human perspective. Live initiates human editable SVGs for both designers and can be used in other applications.
  arXiv  Detail & Related papers  (2022-06-09T17:55:02Z)
• Im2Vec: Synthesizing Vector Graphics without Vector Supervision [31.074606918245298]
  Vector graphics are widely used to represent fonts, logos, digital artworks, and graphic designs. One can alwaysize the input graphic and resort to image-based generative approaches. Current models that require explicit supervision on the vector representation at training time are difficult to obtain. We propose a new neural network that can generate complex vector graphics with varying topologies.
  arXiv  Detail & Related papers  (2021-02-04T18:39:45Z)
• DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation [217.86315551526235]
  We propose a novel hierarchical generative network, called DeepSVG, for complex SVG icons generation and manipulation. Our architecture effectively disentangles high-level shapes from the low-level commands that encode the shape itself. We demonstrate that our network learns to accurately reconstruct diverse vector graphics, and can serve as a powerful animation tool.
  arXiv  Detail & Related papers  (2020-07-22T09:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.