Text2VP: Generative AI for Visual Programming and Parametric Modeling

• URL: http://arxiv.org/abs/2407.07732v1
• Date: Sun, 9 Jun 2024 02:22:20 GMT
• Title: Text2VP: Generative AI for Visual Programming and Parametric Modeling
• Authors: Guangxi Feng, Wei Yan,
• Abstract summary: This study creates and investigates an innovative application of generative AI in parametric modeling by leveraging a customized Text-to-Visual Programming (Text2VP) GPT derived from GPT-4. The primary focus is on automating the generation of graph-based visual programming, including parameters and the links among the parameters, through AI-generated scripts. Our testing demonstrates Text2VP's capability to generate working parametric models.
• Score: 6.531561475204309
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The integration of generative artificial intelligence (AI) into architectural design has witnessed a significant evolution, marked by the recent advancements in AI to generate text, images, and 3D models. However, no models exist for text-to-parametric models that are used in architectural design for generating various design options, including free-form designs, and optimizing the design options. This study creates and investigates an innovative application of generative AI in parametric modeling by leveraging a customized Text-to-Visual Programming (Text2VP) GPT derived from GPT-4. The primary focus is on automating the generation of graph-based visual programming workflows, including parameters and the links among the parameters, through AI-generated scripts, accurately reflecting users' design intentions and allowing the users to change the parameter values interactively. The Text2VP GPT customization process utilizes detailed and complete documentation of the visual programming language components, example-driven few-shot learning, and specific instructional guides. Our testing demonstrates Text2VP's capability to generate working parametric models. The paper also discusses the limitations of Text2VP; for example, more complex parametric model generation introduces higher error rates. This research highlights the potential of generative AI in visual programming and parametric modeling and sets a foundation for future enhancements to handle more sophisticated and intricate modeling tasks effectively. The study aims to allow designers to create and change design models without significant effort in learning a specific programming language such as Grasshopper.

Related papers

• Generative Design through Quality-Diversity Data Synthesis and Language Models [5.196236145367301]
  Two fundamental challenges face generative models in engineering applications: the acquisition of high-performing, diverse datasets, and the adherence to precise constraints in generated designs. We propose a novel approach combining optimization, constraint satisfaction, and language models to tackle these challenges in architectural design.
  arXiv  Detail & Related papers  (2024-05-16T11:30:08Z)
• Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation [87.50120181861362]
  VisionPrefer is a high-quality and fine-grained preference dataset that captures multiple preference aspects. We train a reward model VP-Score over VisionPrefer to guide the training of text-to-image generative models and the preference prediction accuracy of VP-Score is comparable to human annotators.
  arXiv  Detail & Related papers  (2024-04-23T14:53:15Z)
• AI-Generated Images as Data Source: The Dawn of Synthetic Era [61.879821573066216]
  generative AI has unlocked the potential to create synthetic images that closely resemble real-world photographs. This paper explores the innovative concept of harnessing these AI-generated images as new data sources. In contrast to real data, AI-generated data exhibit remarkable advantages, including unmatched abundance and scalability.
  arXiv  Detail & Related papers  (2023-10-03T06:55:19Z)
• InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists [66.85125112199898]
  We develop a unified language interface for computer vision tasks that abstracts away task-specific design choices. Our model, dubbed InstructCV, performs competitively compared to other generalist and task-specific vision models.
  arXiv  Detail & Related papers  (2023-09-30T14:26:43Z)
• RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model [93.8067369210696]
  Text-to-image generation (TTI) refers to the usage of models that could process text input and generate high fidelity images based on text descriptions. Diffusion models are one prominent type of generative model used for the generation of images through the systematic introduction of noises with repeating steps. In the era of large models, scaling up model size and the integration with large language models have further improved the performance of TTI models.
  arXiv  Detail & Related papers  (2023-09-02T03:27:20Z)
• Large Language and Text-to-3D Models for Engineering Design Optimization [0.1740313383876245]
  We study the potential of deep text-to-3D models in the engineering domain. We use Shap-E, a text-to-3D asset network by OpenAI, in the context of aerodynamic vehicle optimization.
  arXiv  Detail & Related papers  (2023-07-03T07:54:09Z)
• Visual Programming for Text-to-Image Generation and Evaluation [73.12069620086311]
  We propose two novel interpretable/explainable visual programming frameworks for text-to-image (T2I) generation and evaluation. First, we introduce VPGen, an interpretable step-by-step T2I generation framework that decomposes T2I generation into three steps: object/count generation, layout generation, and image generation. Second, we introduce VPEval, an interpretable and explainable evaluation framework for T2I generation based on visual programming.
  arXiv  Detail & Related papers  (2023-05-24T16:42:17Z)
• AutoML-GPT: Automatic Machine Learning with GPT [74.30699827690596]
  We propose developing task-oriented prompts and automatically utilizing large language models (LLMs) to automate the training pipeline. We present the AutoML-GPT, which employs GPT as the bridge to diverse AI models and dynamically trains models with optimized hyper parameters. This approach achieves remarkable results in computer vision, natural language processing, and other challenging areas.
  arXiv  Detail & Related papers  (2023-05-04T02:09:43Z)
• Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design [10.531935694354448]
  This paper doubles as a review and practical guide to evaluation metrics for deep generative models (DGMs) in engineering design. We first summarize the well-accepted classic' evaluation metrics for deep generative models grounded in machine learning theory. Next, we curate a set of design-specific metrics which can be used for evaluating deep generative models.
  arXiv  Detail & Related papers  (2023-02-06T16:34:16Z)
• An Overview on Controllable Text Generation via Variational Auto-Encoders [15.97186478109836]
  Recent advances in neural-based generative modeling have reignited the hopes of having computer systems capable of conversing with humans. Latent variable models (LVM) such as variational auto-encoders (VAEs) are designed to characterize the distributional pattern of textual data. This overview gives an introduction to existing generation schemes, problems associated with text variational auto-encoders, and a review of several applications about the controllable generation.
  arXiv  Detail & Related papers  (2022-11-15T07:36:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.