SketchAgent: Generating Structured Diagrams from Hand-Drawn Sketches

• URL: http://arxiv.org/abs/2508.01237v1
• Date: Sat, 02 Aug 2025 07:22:51 GMT
• Title: SketchAgent: Generating Structured Diagrams from Hand-Drawn Sketches
• Authors: Cheng Tan, Qi Chen, Jingxuan Wei, Gaowei Wu, Zhangyang Gao, Siyuan Li, Bihui Yu, Ruifeng Guo, Stan Z. Li,
• Abstract summary: We introduce SketchAgent, a system designed to automate the transformation of hand-drawn sketches into structured diagrams.<n>SketchAgent integrates sketch recognition, symbolic reasoning, and iterative validation to produce semantically coherent and structurally accurate diagrams.<n>By streamlining the diagram generation process, SketchAgent holds great promise for applications in design, education, and engineering.
• Score: 54.06877048295693
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hand-drawn sketches are a natural and efficient medium for capturing and conveying ideas. Despite significant advancements in controllable natural image generation, translating freehand sketches into structured, machine-readable diagrams remains a labor-intensive and predominantly manual task. The primary challenge stems from the inherent ambiguity of sketches, which lack the structural constraints and semantic precision required for automated diagram generation. To address this challenge, we introduce SketchAgent, a multi-agent system designed to automate the transformation of hand-drawn sketches into structured diagrams. SketchAgent integrates sketch recognition, symbolic reasoning, and iterative validation to produce semantically coherent and structurally accurate diagrams, significantly reducing the need for manual effort. To evaluate the effectiveness of our approach, we propose the Sketch2Diagram Benchmark, a comprehensive dataset and evaluation framework encompassing eight diverse diagram categories, such as flowcharts, directed graphs, and model architectures. The dataset comprises over 6,000 high-quality examples with token-level annotations, standardized preprocessing, and rigorous quality control. By streamlining the diagram generation process, SketchAgent holds great promise for applications in design, education, and engineering, while offering a significant step toward bridging the gap between intuitive sketching and machine-readable diagram generation. The benchmark is released at https://huggingface.co/datasets/DiagramAgent/Sketch2Diagram-Benchmark.

Related papers

• SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation [57.47730473674261]
  We introduce SwiftSketch, a model for image-conditioned vector sketch generation that can produce high-quality sketches in less than a second.<n>SwiftSketch operates by progressively denoising stroke control points sampled from a Gaussian distribution.<n>ControlSketch is a method that enhances SDS-based techniques by incorporating precise spatial control through a depth-aware ControlNet.
  arXiv  Detail & Related papers  (2025-02-12T18:57:12Z)
• SketchTriplet: Self-Supervised Scenarized Sketch-Text-Image Triplet Generation [6.39528707908268]
  There continues to be a lack of large-scale paired datasets for scene sketches. We propose a self-supervised method for scene sketch generation that does not rely on any existing scene sketch. We contribute a large-scale dataset centered around scene sketches, comprising highly semantically consistent "text-sketch-image" triplets.
  arXiv  Detail & Related papers  (2024-05-29T06:43:49Z)
• SketchGPT: Autoregressive Modeling for Sketch Generation and Recognition [4.6519578789100215]
  SketchGPT is a flexible framework that employs a sequence-to-sequence autoregressive model for sketch generation, and completion. By mapping complex sketches into simplified sequences of abstract primitives, our approach significantly streamlines the input for autoregressive modeling.
  arXiv  Detail & Related papers  (2024-05-06T01:24:14Z)
• I Know What You Draw: Learning Grasp Detection Conditioned on a Few Freehand Sketches [74.63313641583602]
  We propose a method to generate a potential grasp configuration relevant to the sketch-depicted objects. Our model is trained and tested in an end-to-end manner which is easy to be implemented in real-world applications.
  arXiv  Detail & Related papers  (2022-05-09T04:23:36Z)
• SketchyCOCO: Image Generation from Freehand Scene Sketches [71.85577739612579]
  We introduce the first method for automatic image generation from scene-level freehand sketches. Key contribution is an attribute vector bridged Geneversarative Adrial Network called EdgeGAN. We have built a large-scale composite dataset called SketchyCOCO to support and evaluate the solution.
  arXiv  Detail & Related papers  (2020-03-05T14:54:10Z)
• Sketchformer: Transformer-based Representation for Sketched Structure [12.448155157592895]
  Sketchformer is a transformer-based representation for encoding free-hand sketches input in a vector form. We report several variants exploring continuous and tokenized input representations, and contrast their performance. Our learned embedding, driven by a dictionary learning tokenization scheme, yields state of the art performance in classification and image retrieval tasks.
  arXiv  Detail & Related papers  (2020-02-24T17:11:53Z)
• Deep Self-Supervised Representation Learning for Free-Hand Sketch [51.101565480583304]
  We tackle the problem of self-supervised representation learning for free-hand sketches. Key for the success of our self-supervised learning paradigm lies with our sketch-specific designs. We show that the proposed approach outperforms the state-of-the-art unsupervised representation learning methods.
  arXiv  Detail & Related papers  (2020-02-03T16:28:29Z)
• Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches [133.01690754567252]
  Sketch-based image editing aims to synthesize and modify photos based on the structural information provided by the human-drawn sketches. Deep Plastic Surgery is a novel, robust and controllable image editing framework that allows users to interactively edit images using hand-drawn sketch inputs.
  arXiv  Detail & Related papers  (2020-01-09T08:57:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.