CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM

• URL: http://arxiv.org/abs/2411.04954v1
• Date: Thu, 07 Nov 2024 18:31:08 GMT
• Title: CAD-MLLM: Unifying Multimodality-Conditioned CAD Generation With MLLM
• Authors: Jingwei Xu, Chenyu Wang, Zibo Zhao, Wen Liu, Yi Ma, Shenghua Gao,
• Abstract summary: We introduce the CAD-MLLM, the first system capable of generating parametric CAD models conditioned on the multimodal input. We use advanced large language models (LLMs) to align the feature space across diverse multi-modalities data and CAD models' vectorized representations. Our resulting dataset, named Omni-CAD, is the first multimodal CAD dataset that contains textual description, multi-view images, points, and command sequence for each CAD model.
• Score: 39.113795259823476
• License:
• Abstract: This paper aims to design a unified Computer-Aided Design (CAD) generation system that can easily generate CAD models based on the user's inputs in the form of textual description, images, point clouds, or even a combination of them. Towards this goal, we introduce the CAD-MLLM, the first system capable of generating parametric CAD models conditioned on the multimodal input. Specifically, within the CAD-MLLM framework, we leverage the command sequences of CAD models and then employ advanced large language models (LLMs) to align the feature space across these diverse multi-modalities data and CAD models' vectorized representations. To facilitate the model training, we design a comprehensive data construction and annotation pipeline that equips each CAD model with corresponding multimodal data. Our resulting dataset, named Omni-CAD, is the first multimodal CAD dataset that contains textual description, multi-view images, points, and command sequence for each CAD model. It contains approximately 450K instances and their CAD construction sequences. To thoroughly evaluate the quality of our generated CAD models, we go beyond current evaluation metrics that focus on reconstruction quality by introducing additional metrics that assess topology quality and surface enclosure extent. Extensive experimental results demonstrate that CAD-MLLM significantly outperforms existing conditional generative methods and remains highly robust to noises and missing points. The project page and more visualizations can be found at: https://cad-mllm.github.io/

Related papers

• Img2CAD: Conditioned 3D CAD Model Generation from Single Image with Structured Visual Geometry [12.265852643914439]
  We present Img2CAD, the first knowledge that uses 2D image inputs to generate editable parameters. Img2CAD enables seamless integration between AI 3D reconstruction and CAD representation.
  arXiv  Detail & Related papers  (2024-10-04T13:27:52Z)
• CadVLM: Bridging Language and Vision in the Generation of Parametric CAD Sketches [24.239470848849418]
  Parametric Computer-Aided Design (CAD) is central to contemporary mechanical design. We propose CadVLM, an end-to-end vision language model for CAD generation.
  arXiv  Detail & Related papers  (2024-09-26T01:22:29Z)
• PS-CAD: Local Geometry Guidance via Prompting and Selection for CAD Reconstruction [86.726941702182]
  We introduce geometric guidance into the reconstruction network PS-CAD. We provide the geometry of surfaces where the current reconstruction differs from the complete model as a point cloud. Second, we use geometric analysis to extract a set of planar prompts, that correspond to candidate surfaces.
  arXiv  Detail & Related papers  (2024-05-24T03:43:55Z)
• ContrastCAD: Contrastive Learning-based Representation Learning for Computer-Aided Design Models [0.7373617024876725]
  We propose a contrastive learning-based approach to learning CAD models, named ContrastCAD. ContrastCAD effectively captures semantic information within the construction sequences of the CAD model. We also propose a new CAD data augmentation method, called a Random Replace and Extrude (RRE) method, to enhance the learning performance of the model.
  arXiv  Detail & Related papers  (2024-04-02T05:30:39Z)
• AutoCAD: Automatically Generating Counterfactuals for Mitigating Shortcut Learning [70.70393006697383]
  We present AutoCAD, a fully automatic and task-agnostic CAD generation framework. In this paper, we present AutoCAD, a fully automatic and task-agnostic CAD generation framework.
  arXiv  Detail & Related papers  (2022-11-29T13:39:53Z)
• CADOps-Net: Jointly Learning CAD Operation Types and Steps from Boundary-Representations [17.051792180335354]
  This paper proposes a new deep neural network, CADOps-Net, that jointly learns the CAD operation types and the decomposition into different CAD operation steps. Compared to existing datasets, the complexity and variety of CC3D-Ops models are closer to those used for industrial purposes.
  arXiv  Detail & Related papers  (2022-08-22T19:12:20Z)
• Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image [58.953160501596805]
  We propose a novel approach towards constructing a joint embedding space between 2D images and 3D CAD models in a patch-wise fashion. Our approach is more robust than state of the art in real-world scenarios without any exact CAD matches.
  arXiv  Detail & Related papers  (2021-08-20T20:58:52Z)
• DeepCAD: A Deep Generative Network for Computer-Aided Design Models [37.655225142981564]
  We present the first 3D generative model for a drastically different shape representation -- describing a shape as a sequence of computer-aided design (CAD) operations. Drawing an analogy between CAD operations and natural language, we propose a CAD generative network based on the Transformer.
  arXiv  Detail & Related papers  (2021-05-20T03:29:18Z)
• Fusion 360 Gallery: A Dataset and Environment for Programmatic CAD Construction from Human Design Sequences [43.57844212541765]
  We present the Fusion 360 Gallery, consisting of a simple language with just the sketch and extrude modeling operations. We also present an interactive environment called the Fusion 360 Gym, which exposes the sequential construction of a CAD program as a Markov decision process.
  arXiv  Detail & Related papers  (2020-10-05T23:18:21Z)
• Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve [54.054575408582565]
  We propose to leverage existing large-scale datasets of 3D models to understand the underlying 3D structure of objects seen in an image. We present Mask2CAD, which jointly detects objects in real-world images and for each detected object, optimize for the most similar CAD model and its pose. This produces a clean, lightweight representation of the objects in an image.
  arXiv  Detail & Related papers  (2020-07-26T00:08:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.