Text-Based Reasoning About Vector Graphics
- URL: http://arxiv.org/abs/2404.06479v3
- Date: Fri, 24 May 2024 19:40:26 GMT
- Title: Text-Based Reasoning About Vector Graphics
- Authors: Zhenhailong Wang, Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, Heng Ji,
- Abstract summary: We propose the Visually Descriptive Language Model (VDLM), which performs text-based reasoning about vector graphics.
VDLM bridges with pretrained language models through a newly introduced symbolic representation, Primal Visual Description (PVD)
Our framework offers better interpretability due to its disentangled perception and reasoning processes.
- Score: 76.42082386029206
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While large multimodal models excel in broad vision-language benchmarks, they often struggle with tasks requiring precise perception of low-level visual details, such as comparing line lengths or solving simple mazes. In particular, this failure mode persists in question-answering tasks about vector graphics -- images composed purely of 2D objects and shapes. To address this challenge, we propose the Visually Descriptive Language Model (VDLM), which performs text-based reasoning about vector graphics. VDLM leverages Scalable Vector Graphics (SVG) for a more precise visual description and first uses an off-the-shelf raster-to-SVG algorithm for encoding. Since existing language models cannot understand raw SVGs in a zero-shot setting, VDLM then bridges SVG with pretrained language models through a newly introduced intermediate symbolic representation, Primal Visual Description (PVD), comprising primitive attributes (e.g., shape, position, measurement) with their corresponding predicted values. PVD is task-agnostic and represents visual primitives that are universal across all vector graphics. It can be learned with procedurally generated (SVG, PVD) pairs and also enables the direct use of LLMs for generalization to complex reasoning tasks. By casting an image to a text-based representation, we can leverage the power of language models to learn alignment from SVG to visual primitives and generalize to unseen question-answering tasks. Empirical results show that VDLM achieves stronger zero-shot performance compared to state-of-the-art LMMs, such as GPT-4V, in various low-level multimodal perception and reasoning tasks on vector graphics. We additionally present extensive analyses on VDLM's performance, demonstrating that our framework offers better interpretability due to its disentangled perception and reasoning processes. Project page: https://mikewangwzhl.github.io/VDLM/
Related papers
- VGBench: Evaluating Large Language Models on Vector Graphics Understanding and Generation [28.1277394934428]
VGBench is a comprehensive benchmark for Large Language Models (LLMs) on handling vector graphics.
LLMs show strong capability on both aspects while exhibiting less desirable performance on low-level formats (SVG)
arXiv Detail & Related papers (2024-07-15T17:59:55Z) - SuperSVG: Superpixel-based Scalable Vector Graphics Synthesis [66.44553285020066]
SuperSVG is a superpixel-based vectorization model that achieves fast and high-precision image vectorization.
We propose a two-stage self-training framework, where a coarse-stage model is employed to reconstruct the main structure and a refinement-stage model is used for enriching the details.
Experiments demonstrate the superior performance of our method in terms of reconstruction accuracy and inference time compared to state-of-the-art approaches.
arXiv Detail & Related papers (2024-06-14T07:43:23Z) - StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis [112.25071764647683]
StrokeNUWA is a pioneering work exploring a better visual representation ''stroke tokens'' on vector graphics.
equipped with stroke tokens, StrokeNUWA can significantly surpass traditional LLM-based and optimization-based methods.
StrokeNUWA achieves up to a 94x speedup in inference over the speed of prior methods with an exceptional SVG code compression ratio of 6.9%.
arXiv Detail & Related papers (2024-01-30T15:20:26Z) - SVGDreamer: Text Guided SVG Generation with Diffusion Model [31.76771064173087]
We propose a novel text-guided vector graphics synthesis method called SVGDreamer.
SIVE process enables decomposition of synthesis into foreground objects and background.
VPSD approach addresses issues of shape over-smoothing, color over-saturation, limited diversity, and slow convergence.
arXiv Detail & Related papers (2023-12-27T08:50:01Z) - Beyond Pixels: Exploring Human-Readable SVG Generation for Simple Images
with Vision Language Models [19.145503353922038]
We introduce our method, Simple-SVG-Generation (Stextsuperscript2VGtextsuperscript2).
Our method focuses on producing SVGs that are both accurate and simple, aligning with human readability and understanding.
With simple images, we evaluate our method with reasoning tasks together with advanced language models, the results show a clear improvement over previous SVG generation methods.
arXiv Detail & Related papers (2023-11-27T05:20:11Z) - SAMVG: A Multi-stage Image Vectorization Model with the Segment-Anything
Model [59.40189857428461]
We propose a multi-stage model to vectorize images into SVG (Scalable Vector Graphics)
Firstly, SAMVG uses general image segmentation provided by the Segment-Anything Model and uses a novel filtering method to identify the best dense segmentation map for the entire image.
Secondly, SAMVG then identifies missing components and adds more detailed components to the SVG.
arXiv Detail & Related papers (2023-11-09T11:11:56Z) - VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models [82.93345261434943]
We show that a text-conditioned diffusion model trained on pixel representations of images can be used to generate SVG-exportable vector graphics.
Inspired by recent text-to-3D work, we learn an SVG consistent with a caption using Score Distillation Sampling.
Experiments show greater quality than prior work, and demonstrate a range of styles including pixel art and sketches.
arXiv Detail & Related papers (2022-11-21T10:04:27Z) - Towards Layer-wise Image Vectorization [57.26058135389497]
We propose Layerwise Image Vectorization, namely LIVE, to convert images to SVGs and simultaneously maintain its image topology.
Live generates compact forms with layer-wise structures that are semantically consistent with human perspective.
Live initiates human editable SVGs for both designers and can be used in other applications.
arXiv Detail & Related papers (2022-06-09T17:55:02Z) - Im2Vec: Synthesizing Vector Graphics without Vector Supervision [31.074606918245298]
Vector graphics are widely used to represent fonts, logos, digital artworks, and graphic designs.
One can alwaysize the input graphic and resort to image-based generative approaches.
Current models that require explicit supervision on the vector representation at training time are difficult to obtain.
We propose a new neural network that can generate complex vector graphics with varying topologies.
arXiv Detail & Related papers (2021-02-04T18:39:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.