Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AI

• URL: http://arxiv.org/abs/2405.12985v1
• Date: Tue, 26 Mar 2024 02:12:17 GMT
• Title: Sketch2Prototype: Rapid Conceptual Design Exploration and Prototyping with Generative AI
• Authors: Kristen M. Edwards, Brandon Man, Faez Ahmed,
• Abstract summary: Sketch2Prototype is an AI-based framework that transforms a hand-drawn sketch into a diverse set of 2D images and 3D prototypes. We show that using text as an intermediate modality outperforms direct sketch-to-3D baselines for generating diverse and manufacturable 3D models.
• Score: 3.936104238911733
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sketch2Prototype is an AI-based framework that transforms a hand-drawn sketch into a diverse set of 2D images and 3D prototypes through sketch-to-text, text-to-image, and image-to-3D stages. This framework, shown across various sketches, rapidly generates text, image, and 3D modalities for enhanced early-stage design exploration. We show that using text as an intermediate modality outperforms direct sketch-to-3D baselines for generating diverse and manufacturable 3D models. We find limitations in current image-to-3D techniques, while noting the value of the text modality for user-feedback and iterative design augmentation.

Related papers

• Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering [17.918603435615335]
  3D sketches are widely used for visually representing the 3D shape and structure of objects or scenes. We propose Diff3DS, a novel differentiable framework for generating view-consistent 3D sketch. Our framework bridges the domains of 3D sketch and customized image, achieving end-toend optimization of 3D sketch.
  arXiv  Detail & Related papers  (2024-05-24T07:48:14Z)
• Sketch3D: Style-Consistent Guidance for Sketch-to-3D Generation [55.73399465968594]
  This paper proposes a novel generation paradigm Sketch3D to generate realistic 3D assets with shape aligned with the input sketch and color matching the textual description. Three strategies are designed to optimize 3D Gaussians, i.e., structural optimization via a distribution transfer mechanism, color optimization with a straightforward MSE loss and sketch similarity optimization with a CLIP-based geometric similarity loss.
  arXiv  Detail & Related papers  (2024-04-02T11:03:24Z)
• Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation [37.93542778715304]
  We present a sketch-guided text-to-3D generation framework (namely, Sketch2NeRF) to add sketch control to 3D generation. Our method achieves state-of-the-art performance in terms of sketch similarity and text alignment.
  arXiv  Detail & Related papers  (2024-01-25T15:49:12Z)
• Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes [118.406721663244]
  We introduce a novel part-level modelling and alignment framework that facilitates abstraction modelling and cross-modal correspondence. Our approach seamlessly extends to sketch modelling by establishing correspondence between CLIPasso edgemaps and projected 3D part regions.
  arXiv  Detail & Related papers  (2023-12-07T05:04:33Z)
• Neural 3D Strokes: Creating Stylized 3D Scenes with Vectorized 3D Strokes [20.340259111585873]
  We present Neural 3D Strokes, a novel technique to generate stylized images of a 3D scene at arbitrary novel views from multi-view 2D images. Our approach draws inspiration from image-to-painting methods, simulating the progressive painting process of human artwork with vector strokes.
  arXiv  Detail & Related papers  (2023-11-27T09:02:21Z)
• Control3D: Towards Controllable Text-to-3D Generation [107.81136630589263]
  We present a text-to-3D generation conditioning on the additional hand-drawn sketch, namely Control3D. A 2D conditioned diffusion model (ControlNet) is remoulded to guide the learning of 3D scene parameterized as NeRF. We exploit a pre-trained differentiable photo-to-sketch model to directly estimate the sketch of the rendered image over synthetic 3D scene.
  arXiv  Detail & Related papers  (2023-11-09T15:50:32Z)
• Guide3D: Create 3D Avatars from Text and Image Guidance [55.71306021041785]
  Guide3D is a text-and-image-guided generative model for 3D avatar generation based on diffusion models. Our framework produces topologically and structurally correct geometry and high-resolution textures.
  arXiv  Detail & Related papers  (2023-08-18T17:55:47Z)
• TAPS3D: Text-Guided 3D Textured Shape Generation from Pseudo Supervision [114.56048848216254]
  We present a novel framework, TAPS3D, to train a text-guided 3D shape generator with pseudo captions. Based on rendered 2D images, we retrieve relevant words from the CLIP vocabulary and construct pseudo captions using templates. Our constructed captions provide high-level semantic supervision for generated 3D shapes.
  arXiv  Detail & Related papers  (2023-03-23T13:53:16Z)
• Make Your Brief Stroke Real and Stereoscopic: 3D-Aware Simplified Sketch to Portrait Generation [51.64832538714455]
  Existing studies only generate portraits in the 2D plane with fixed views, making the results less vivid. In this paper, we present Stereoscopic Simplified Sketch-to-Portrait (SSSP), which explores the possibility of creating Stereoscopic 3D-aware portraits. Our key insight is to design sketch-aware constraints that can fully exploit the prior knowledge of a tri-plane-based 3D-aware generative model.
  arXiv  Detail & Related papers  (2023-02-14T06:28:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.