CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches

• URL: http://arxiv.org/abs/2409.17457v1
• Date: Thu, 26 Sep 2024 01:22:29 GMT
• Title: CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches
• Authors: Sifan Wu, Amir Khasahmadi, Mor Katz, Pradeep Kumar Jayaraman, Yewen Pu, Karl Willis, Bang Liu,
• Abstract summary: Parametric Computer-Aided Design (CAD) is central to contemporary mechanical design. We propose CadVLM, an end-to-end vision language model for CAD generation.
• Score: 24.239470848849418
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Parametric Computer-Aided Design (CAD) is central to contemporary mechanical design. However, it encounters challenges in achieving precise parametric sketch modeling and lacks practical evaluation metrics suitable for mechanical design. We harness the capabilities of pre-trained foundation models, renowned for their successes in natural language processing and computer vision, to develop generative models specifically for CAD. These models are adept at understanding complex geometries and design reasoning, a crucial advancement in CAD technology. In this paper, we propose CadVLM, an end-to-end vision language model for CAD generation. Our approach involves adapting pre-trained foundation models to manipulate engineering sketches effectively, integrating both sketch primitive sequences and sketch images. Extensive experiments demonstrate superior performance on multiple CAD sketch generation tasks such as CAD autocompletion, CAD autoconstraint, and image conditional generation. To our knowledge, this is the first instance of a multimodal Large Language Model (LLM) being successfully applied to parametric CAD generation, representing a pioneering step in the field of computer-aided mechanical design.

Related papers

• Text2CAD: Text to 3D CAD Generation via Technical Drawings [45.3611544056261]
  Text2CAD is a novel framework that employs stable diffusion models tailored to automate the generation process. We show that Text2CAD effectively generates technical drawings that are accurately translated into high-quality 3D CAD models.
  arXiv  Detail & Related papers  (2024-11-09T15:12:06Z)
• PS-CAD: Local Geometry Guidance via Prompting and Selection for CAD Reconstruction [86.726941702182]
  We introduce geometric guidance into the reconstruction network PS-CAD. We provide the geometry of surfaces where the current reconstruction differs from the complete model as a point cloud. Second, we use geometric analysis to extract a set of planar prompts, that correspond to candidate surfaces.
  arXiv  Detail & Related papers  (2024-05-24T03:43:55Z)
• Geometric Deep Learning for Computer-Aided Design: A Survey [85.79012726689511]
  This survey offers a comprehensive overview of learning-based methods in computer-aided design. It includes similarity analysis and retrieval, 2D and 3D CAD model synthesis, and CAD generation from point clouds. It provides a complete list of benchmark datasets and their characteristics, along with open-source codes that have propelled research in this domain.
  arXiv  Detail & Related papers  (2024-02-27T17:11:35Z)
• CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention [13.227571488321358]
  We propose an end-to-end trainable and auto-regressive architecture to recover the design history of a CAD model. Our model learns visual-language representations by layer-wise cross-attention between point cloud and CAD language embedding. Thanks to its auto-regressive nature, CAD-SIGNet not only reconstructs a unique full design history of the corresponding CAD model given an input point cloud but also provides multiple plausible design choices.
  arXiv  Detail & Related papers  (2024-02-27T16:53:16Z)
• PPI-NET: End-to-End Parametric Primitive Inference [24.31083483088741]
  In engineering applications, line, circle, arc, and point are collectively referred to as primitives. We propose an efficient and accurate end-to-end method to infer parametric primitives from hand-drawn sketch images.
  arXiv  Detail & Related papers  (2023-08-03T03:50:49Z)
• AutoCAD: Automatically Generating Counterfactuals for Mitigating Shortcut Learning [70.70393006697383]
  We present AutoCAD, a fully automatic and task-agnostic CAD generation framework. In this paper, we present AutoCAD, a fully automatic and task-agnostic CAD generation framework.
  arXiv  Detail & Related papers  (2022-11-29T13:39:53Z)
• Vitruvion: A Generative Model of Parametric CAD Sketches [22.65229769427499]
  We present an approach to generative modeling of parametric CAD sketches. Our model, trained on real-world designs from the SketchGraphs dataset, autoregressively synthesizes sketches as sequences of primitives. We condition the model on various contexts, including partial sketches (primers) and images of hand-drawn sketches.
  arXiv  Detail & Related papers  (2021-09-29T01:02:30Z)
• Using Machine Learning to Predict Engineering Technology Students' Success with Computer Aided Design [50.591267188664666]
  We show how data combined with machine learning techniques can predict how well a particular student will perform in a design task. We found that our models using early design sequence actions are particularly valuable for prediction. Further improvements to these models could lead to earlier predictions and thus provide students feedback sooner to enhance their learning.
  arXiv  Detail & Related papers  (2021-08-12T20:24:54Z)
• Engineering Sketch Generation for Computer-Aided Design [10.732102570751392]
  We propose two generative models, CurveGen and TurtleGen, for engineering sketch generation. Both models generate curve primitives without the need for a sketch constraint solver. We find in our perceptual evaluation using human subjects that both CurveGen and TurtleGen produce more realistic engineering sketches.
  arXiv  Detail & Related papers  (2021-04-19T20:38:36Z)
• Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequences [43.57844212541765]
  We present the Fusion 360 Gallery, consisting of a simple language with just the sketch and extrude modeling operations. We also present an interactive environment called the Fusion 360 Gym, which exposes the sequential construction of a CAD program as a Markov decision process.
  arXiv  Detail & Related papers  (2020-10-05T23:18:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.