LLM2FEA: Discover Novel Designs with Generative Evolutionary Multitasking

• URL: http://arxiv.org/abs/2406.14917v1
• Date: Fri, 21 Jun 2024 07:20:51 GMT
• Title: LLM2FEA: Discover Novel Designs with Generative Evolutionary Multitasking
• Authors: Melvin Wong, Jiao Liu, Thiago Rios, Stefan Menzel, Yew Soon Ong,
• Abstract summary: We propose the first attempt to discover novel designs in generative models by transferring knowledge across multiple domains. By utilizing a multi-factorial evolutionary algorithm (MFEA) to drive a large language model, LLM2FEA integrates knowledge from various fields to generate prompts that guide the generative model in discovering novel and practical objects.
• Score: 21.237950330178354
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid research and development of generative artificial intelligence has enabled the generation of high-quality images, text, and 3D models from text prompts. This advancement impels an inquiry into whether these models can be leveraged to create digital artifacts for both creative and engineering applications. Drawing on innovative designs from other domains may be one answer to this question, much like the historical practice of ``bionics", where humans have sought inspiration from nature's exemplary designs. This raises the intriguing possibility of using generative models to simultaneously tackle design tasks across multiple domains, facilitating cross-domain learning and resulting in a series of innovative design solutions. In this paper, we propose LLM2FEA as the first attempt to discover novel designs in generative models by transferring knowledge across multiple domains. By utilizing a multi-factorial evolutionary algorithm (MFEA) to drive a large language model, LLM2FEA integrates knowledge from various fields to generate prompts that guide the generative model in discovering novel and practical objects. Experimental results in the context of 3D aerodynamic design verify the discovery capabilities of the proposed LLM2FEA. The designs generated by LLM2FEA not only satisfy practicality requirements to a certain degree but also feature novel and aesthetically pleasing shapes, demonstrating the potential applications of LLM2FEA in discovery tasks.

Related papers

• Grammar-Guided Evolutionary Search for Discrete Prompt Optimisation [63.97051732013936]
  We propose an evolutionary search approach to automated discrete prompt optimisation consisting of two phases.<n>In the first phase, grammar-guided genetic programming is invoked to synthesise prompt-creating programmes.<n>In the second phase, local search is applied to explore the neighbourhoods of best-performing programmes.
  arXiv  Detail & Related papers  (2025-07-14T14:34:15Z)
• LLM-to-Phy3D: Physically Conform Online 3D Object Generation with LLMs [25.95070778191463]
  We introduce LLM-to-Phy3D, a physically conform online 3D object generation that enables existing LLM-to-3D models to produce conforming 3D objects on the fly.<n> Systematic evaluations of LLM-to-Phy3D, supported by ablation studies in vehicle design optimization, reveal various LLM improvements gained by 4.5% to 106.7%.<n>The encouraging results suggest the potential general use of LLM-to-Phy3D in Physical AI for scientific and engineering applications.
  arXiv  Detail & Related papers  (2025-06-11T10:06:21Z)
• Reinforcing Multimodal Understanding and Generation with Dual Self-rewards [56.08202047680044]
  Large language models (LLMs) unify cross-model understanding and generation into a single framework.<n>Current solutions require external supervision (e.g., human feedback or reward models) and only address unidirectional tasks.<n>We introduce a self-supervised dual reward mechanism to reinforce the understanding and generation capabilities of LMMs.
  arXiv  Detail & Related papers  (2025-06-09T17:38:45Z)
• Cooking Up Creativity: A Cognitively-Inspired Approach for Enhancing LLM Creativity through Structured Representations [53.950760059792614]
  Large Language Models (LLMs) excel at countless tasks, yet struggle with creativity. We introduce a novel approach that couples LLMs with structured representations and cognitively inspired manipulations to generate more creative and diverse ideas. We demonstrate our approach in the culinary domain with DishCOVER, a model that generates creative recipes.
  arXiv  Detail & Related papers  (2025-04-29T11:13:06Z)
• How Effective are Generative Large Language Models in Performing Requirements Classification? [4.429729688079712]
  This study explores the effectiveness of three generative large language models (LLMs) performing both binary and multi-class requirements classification.<n>Our study concludes that while factors like prompt design and LLM architecture are universally important, others-such as dataset variations-have a more situational impact, depending on the complexity of the classification task.
  arXiv  Detail & Related papers  (2025-04-23T14:41:11Z)
• Probing and Inducing Combinational Creativity in Vision-Language Models [52.76981145923602]
  Recent advances in Vision-Language Models (VLMs) have sparked debate about whether their outputs reflect combinational creativity. We propose the Identification-Explanation-Implication (IEI) framework, which decomposes creative processes into three levels. To validate this framework, we curate CreativeMashup, a high-quality dataset of 666 artist-generated visual mashups annotated according to the IEI framework.
  arXiv  Detail & Related papers  (2025-04-17T17:38:18Z)
• Aligning Large Language Models and Geometric Deep Models for Protein Representation [57.59506688299817]
  Latent representation alignment is used to map embeddings from different modalities into a shared space, often aligned with the embedding space of large language models (LLMs)<n>Preliminary protein-focused large language models (MLLMs) have emerged, but they have predominantly relied on approaches lacking a fundamental understanding of optimal alignment practices across representations.<n>In this study, we explore the alignment of multimodal representations between LLMs and Geometric Deep Models (GDMs) in the protein domain.<n>Our work examines alignment factors from both model and protein perspectives, identifying challenges in current alignment methodologies and proposing strategies to improve the alignment process.
  arXiv  Detail & Related papers  (2024-11-08T04:15:08Z)
• EMMA: Efficient Visual Alignment in Multi-Modal LLMs [56.03417732498859]
  EMMA is a lightweight cross-modality module designed to efficiently fuse visual and textual encodings.<n>EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations.
  arXiv  Detail & Related papers  (2024-10-02T23:00:31Z)
• Multi-Modal Generative AI: Multi-modal LLM, Diffusion and Beyond [48.43910061720815]
  Multi-modal generative AI has received increasing attention in both academia and industry. One natural question arises: Is it possible to have a unified model for both understanding and generation?
  arXiv  Detail & Related papers  (2024-09-23T13:16:09Z)
• Benchmarking Language Model Creativity: A Case Study on Code Generation [39.546827184857754]
  In this work, we introduce a framework for quantifying LLM creativity. We define NEOGAUGE, a metric that quantifies both convergent and divergent thinking in the generated creative responses. We test the proposed framework on Codeforces problems, which serve as both a natural dataset for coding tasks and a collection of prior human solutions.
  arXiv  Detail & Related papers  (2024-07-12T05:55:22Z)
• Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks [22.66167973623777]
  Large Language Models (LLMs) have transformed NLP with their remarkable In-context Learning (ICL) capabilities. This paper investigates whether LLMs can generalize from labeled examples of predefined tasks to novel tasks. We show that cross-task prompting leads to a remarkable performance boost of 107% for LLaMA-2 7B, 18.6% for LLaMA-2 13B, and 3.2% for GPT 3.5 on average over zero-shot prompting.
  arXiv  Detail & Related papers  (2024-05-17T05:20:49Z)
• Geometric Deep Learning for Computer-Aided Design: A Survey [85.79012726689511]
  This survey offers a comprehensive overview of learning-based methods in computer-aided design. It includes similarity analysis and retrieval, 2D and 3D CAD model synthesis, and CAD generation from point clouds. It provides a complete list of benchmark datasets and their characteristics, along with open-source codes that have propelled research in this domain.
  arXiv  Detail & Related papers  (2024-02-27T17:11:35Z)
• When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challenges [50.280704114978384]
  Pre-trained large language models (LLMs) exhibit powerful capabilities for generating natural text.<n> Evolutionary algorithms (EAs) can discover diverse solutions to complex real-world problems.
  arXiv  Detail & Related papers  (2024-01-19T05:58:30Z)
• 3DGEN: A GAN-based approach for generating novel 3D models from image data [5.767281919406463]
  We present 3DGEN, a model that leverages the recent work on both Neural Radiance Fields for object reconstruction and GAN-based image generation. We show that the proposed architecture can generate plausible meshes for objects of the same category as the training images and compare the resulting meshes with the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-12-13T12:24:34Z)
• DreamCreature: Crafting Photorealistic Virtual Creatures from Imagination [140.1641573781066]
  We introduce a novel task, Virtual Creatures Generation: Given a set of unlabeled images of the target concepts, we aim to train a T2I model capable of creating new, hybrid concepts. We propose a new method called DreamCreature, which identifies and extracts the underlying sub-concepts. The T2I thus adapts to generate novel concepts with faithful structures and photorealistic appearance.
  arXiv  Detail & Related papers  (2023-11-27T01:24:31Z)
• Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation [19.62178304006683]
  We argue that current interaction paradigms fall short, guiding users towards rapid convergence on a limited set of ideas. We propose a framework that facilitates the structured generation of design space in which users can seamlessly explore, evaluate, and synthesize a multitude of responses.
  arXiv  Detail & Related papers  (2023-10-19T17:53:14Z)
• State of the Art on Diffusion Models for Visual Computing [191.6168813012954]
  This report introduces the basic mathematical concepts of diffusion models, implementation details and design choices of the popular Stable Diffusion model. We also give a comprehensive overview of the rapidly growing literature on diffusion-based generation and editing. We discuss available datasets, metrics, open challenges, and social implications.
  arXiv  Detail & Related papers  (2023-10-11T05:32:29Z)
• Making LLaMA SEE and Draw with SEED Tokenizer [69.1083058794092]
  We introduce SEED, an elaborate image tokenizer that empowers Large Language Models with the ability to SEE and Draw. With SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe. SEED-LLaMA has exhibited compositional emergent abilities such as multi-turn in-context multimodal generation.
  arXiv  Detail & Related papers  (2023-10-02T14:03:02Z)
• Breathing New Life into 3D Assets with Generative Repainting [74.80184575267106]
  Diffusion-based text-to-image models ignited immense attention from the vision community, artists, and content creators. Recent works have proposed various pipelines powered by the entanglement of diffusion models and neural fields. We explore the power of pretrained 2D diffusion models and standard 3D neural radiance fields as independent, standalone tools. Our pipeline accepts any legacy renderable geometry, such as textured or untextured meshes, and orchestrates the interaction between 2D generative refinement and 3D consistency enforcement tools.
  arXiv  Detail & Related papers  (2023-09-15T16:34:51Z)
• Large Language and Text-to-3D Models for Engineering Design Optimization [0.1740313383876245]
  We study the potential of deep text-to-3D models in the engineering domain. We use Shap-E, a text-to-3D asset network by OpenAI, in the context of aerodynamic vehicle optimization.
  arXiv  Detail & Related papers  (2023-07-03T07:54:09Z)
• MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks [59.09343552273045]
  We propose a decoder-only model for multimodal tasks, which is surprisingly effective in jointly learning of these disparate vision-language tasks. We demonstrate that joint learning of these diverse objectives is simple, effective, and maximizes the weight-sharing of the model across these tasks. Our model achieves the state of the art on image-text and text-image retrieval, video question answering and open-vocabulary detection tasks, outperforming much larger and more extensively trained foundational models.
  arXiv  Detail & Related papers  (2023-03-29T16:42:30Z)
• Investigating GANsformer: A Replication Study of a State-of-the-Art Image Generation Model [0.0]
  We reproduce and evaluate a novel variation of the original GAN network, the GANformer. Due to resources and time limitations, we had to constrain the network's training times, dataset types, and sizes.
  arXiv  Detail & Related papers  (2023-03-15T12:51:16Z)
• Challenges in creative generative models for music: a divergence maximization perspective [3.655021726150369]
  Development of generative Machine Learning models in creative practices is raising more interest among artists, practitioners and performers. Most models are still unable to generate content that lay outside of the domain defined by the training dataset. We propose an alternative prospective framework, starting from a new general formulation of ML objectives.
  arXiv  Detail & Related papers  (2022-11-16T12:02:43Z)
• CreativeGAN: Editing Generative Adversarial Networks for Creative Design Synthesis [1.933681537640272]
  This paper proposes an automated method, named CreativeGAN, for generating novel designs. It does so by identifying components that make a design unique and modifying a GAN model such that it becomes more likely to generate designs with identified unique components. Using a dataset of bicycle designs, we demonstrate that the method can create new bicycle designs with unique frames and handles, and rare novelties to a broad set of designs.
  arXiv  Detail & Related papers  (2021-03-10T18:22:35Z)
• MO-PaDGAN: Generating Diverse Designs with Multivariate Performance Enhancement [13.866787416457454]
  Deep generative models have proven useful for automatic design synthesis and design space exploration. They face three challenges when applied to engineering design: 1) generated designs lack diversity, 2) it is difficult to explicitly improve all the performance measures of generated designs, and 3) existing models generally do not generate high-performance novel designs. We propose MO-PaDGAN, which contains a new Determinantal Point Processes based loss function for probabilistic modeling of diversity and performances.
  arXiv  Detail & Related papers  (2020-07-07T21:57:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.