Empowering LLMs to Understand and Generate Complex Vector Graphics

• URL: http://arxiv.org/abs/2412.11102v2
• Date: Wed, 08 Jan 2025 07:22:51 GMT
• Title: Empowering LLMs to Understand and Generate Complex Vector Graphics
• Authors: Ximing Xing, Juncheng Hu, Guotao Liang, Jing Zhang, Dong Xu, Qian Yu,
• Abstract summary: Large Language Models (LLMs) encode partial knowledge of vector graphics from web pages during training.<n>Recent findings suggest that semantically ambiguous and tokenized representations within LLMs may result in hallucinations in vector primitive predictions.<n>We present LLM4SVG, an initial yet substantial step toward bridging this gap by enabling LLMs to better understand and generate vector graphics.
• Score: 30.21003939248769
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The unprecedented advancements in Large Language Models (LLMs) have profoundly impacted natural language processing but have yet to fully embrace the realm of scalable vector graphics (SVG) generation. While LLMs encode partial knowledge of SVG data from web pages during training, recent findings suggest that semantically ambiguous and tokenized representations within LLMs may result in hallucinations in vector primitive predictions. Additionally, LLM training typically lacks modeling and understanding of the rendering sequence of vector paths, which can lead to occlusion between output vector primitives. In this paper, we present LLM4SVG, an initial yet substantial step toward bridging this gap by enabling LLMs to better understand and generate vector graphics. LLM4SVG facilitates a deeper understanding of SVG components through learnable semantic tokens, which precisely encode these tokens and their corresponding properties to generate semantically aligned SVG outputs. Using a series of learnable semantic tokens, a structured dataset for instruction following is developed to support comprehension and generation across two primary tasks. Our method introduces a modular architecture to existing large language models, integrating semantic tags, vector instruction encoders, fine-tuned commands, and powerful LLMs to tightly combine geometric, appearance, and language information. To overcome the scarcity of SVG-text instruction data, we developed an automated data generation pipeline that collected a massive dataset of more than 250k SVG data and 580k SVG-text instructions, which facilitated the adoption of the two-stage training strategy popular in LLM development. By exploring various training strategies, we developed LLM4SVG, which significantly moves beyond optimized rendering-based approaches and language-model-based baselines to achieve remarkable results in human evaluation tasks.

Related papers

• OmniSVG: A Unified Scalable Vector Graphics Generation Model [70.26163703054979]
  We propose OmniSVG, a unified framework that leverages pre-trained Vision-Language Models for end-to-end multimodal SVG generation. By parameterizing SVG commands and coordinates into discrete tokens, OmniSVG decouples structural logic from low-level geometry for efficient training while maintaining the synthesis of complex SVG structure. We introduce MMSVG-2M, a multimodal dataset with two million annotated SVG assets, along with a standardized evaluation protocol for conditional SVG generation tasks.
  arXiv  Detail & Related papers  (2025-04-08T17:59:49Z)
• From Text to Visuals: Using LLMs to Generate Math Diagrams with Vector Graphics [4.012351415340318]
  Large language models (LLMs) offer new possibilities for enhancing math education by automating support for both teachers and students. Recent research on using LLMs to generate Scalable Vector Graphics (SVG) presents a promising approach to automating diagram creation. This paper addresses three research questions: (1) how to automatically generate math diagrams in problem-solving hints and evaluate their quality, (2) whether SVG is an effective intermediate representation for math diagrams, and (3) what prompting strategies and formats are required for LLMs to generate accurate SVG-based diagrams.
  arXiv  Detail & Related papers  (2025-03-10T15:13:38Z)
• Leveraging Large Language Models For Scalable Vector Graphics Processing: A Review [0.0]
  Traditional vectorization techniques suffer from long processing times and excessive output complexity. The advent of large language models (LLMs) has opened new possibilities for the generation, editing, and analysis of vector graphics.
  arXiv  Detail & Related papers  (2025-03-06T21:23:17Z)
• NeuralSVG: An Implicit Representation for Text-to-Vector Generation [54.4153300455889]
  We propose NeuralSVG, an implicit neural representation for generating vector graphics from text prompts. To encourage a layered structure in the generated SVG, we introduce a dropout-based regularization technique. We demonstrate that NeuralSVG outperforms existing methods in generating structured and flexible SVG.
  arXiv  Detail & Related papers  (2025-01-07T18:50:06Z)
• GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model [63.774726052837266]
  We introduce a new architecture that deeply integrates Graph Neural Networks (GNNs) with Large Language Models (LLMs) We introduce three key innovations: (1) Structure-Aware Transformers, which incorporate GNN's message-passing capabilities directly into LLM's transformer layers; (2) Graph-Text Cross-Attention, which processes full, uncompressed text from graph nodes and edges; and (3) GNN-LLM Twin Predictor, enabling LLM's flexible autoregressive generation alongside GNN's scalable one-pass prediction.
  arXiv  Detail & Related papers  (2024-12-08T05:49:58Z)
• Chat2SVG: Vector Graphics Generation with Large Language Models and Image Diffusion Models [14.917583676464266]
  Chat2SVG is a hybrid framework that combines Large Language Models and image diffusion models for text-to-SVG generation. Our system enables intuitive editing through natural language instructions, making professional vector graphics creation accessible to all users.
  arXiv  Detail & Related papers  (2024-11-25T17:31:57Z)
• LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models [62.85566496673856]
  This work explores expanding the capabilities of large language models (LLMs) pretrained on text to generate 3D meshes within a unified model. A primary challenge is effectively tokenizing 3D mesh data into discrete tokens that LLMs can process seamlessly. Our work is the first to demonstrate that LLMs can be fine-tuned to acquire complex spatial knowledge for 3D mesh generation in a text-based format.
  arXiv  Detail & Related papers  (2024-11-14T17:08:23Z)
• VP-LLM: Text-Driven 3D Volume Completion with Large Language Models through Patchification [56.211321810408194]
  Large language models (LLMs) have shown great potential in multi-modal understanding and generation tasks. We present Volume Patch LLM (VP-LLM), which leverages LLMs to perform conditional 3D completion in a single-forward pass. Our results demonstrate a strong ability of LLMs to interpret complex text instructions and understand 3D objects, surpassing state-of-the-art diffusion-based 3D completion models in generation quality.
  arXiv  Detail & Related papers  (2024-06-08T18:17:09Z)
• Visually Descriptive Language Model for Vector Graphics Reasoning [76.42082386029206]
  We propose the Visually Descriptive Language Model (VDLM) to bridge the gap between low-level visual perception and high-level language reasoning. We show that VDLM significantly improves state-of-the-art LMMs like GPT-4o on various multimodal perception and reasoning tasks.
  arXiv  Detail & Related papers  (2024-04-09T17:30:18Z)
• Leveraging Large Language Models for Scalable Vector Graphics-Driven Image Understanding [46.042197741423365]
  Large language models (LLMs) have made significant advancements in natural language understanding. This work investigates if it is possible for the LLM to understand images as well.
  arXiv  Detail & Related papers  (2023-06-09T17:57:01Z)
• Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning [51.90524745663737]
  A key innovation is our use of explanations as features, which can be used to boost GNN performance on downstream tasks. Our method achieves state-of-the-art results on well-established TAG datasets. Our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv.
  arXiv  Detail & Related papers  (2023-05-31T03:18:03Z)
• DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation [217.86315551526235]
  We propose a novel hierarchical generative network, called DeepSVG, for complex SVG icons generation and manipulation. Our architecture effectively disentangles high-level shapes from the low-level commands that encode the shape itself. We demonstrate that our network learns to accurately reconstruct diverse vector graphics, and can serve as a powerful animation tool.
  arXiv  Detail & Related papers  (2020-07-22T09:36:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.