Text-to-Image Generative AI for Modeling and Simulation: Methods, Opportunities, and Applications

• URL: http://arxiv.org/abs/2606.20991v1
• Date: Thu, 18 Jun 2026 23:48:32 GMT
• Title: Text-to-Image Generative AI for Modeling and Simulation: Methods, Opportunities, and Applications
• Abstract summary: Text-to-image generation is a form of generative artificial intelligence (GenAI) that converts textual descriptions into images.<n>This tutorial introduces text-to-image generation to the modeling and simulation (M&S) community.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Text-to-image generation is a form of generative artificial intelligence (GenAI) that converts textual descriptions into images. Most applications of GenAI in modeling and simulation (M&S) have focused on large language models for documentation, coding, or explanation. By contrast, the potential of image generation remains largely unexplored. This tutorial introduces text-to-image generation to the M&S community and details how it can support several M&S tasks, including communicating conceptual models, visualizing simulation outcomes, generating educational materials, and interfacing heterogeneous models in multi-scale simulations. The tutorial combines conceptual guidance with practical workflows, explaining how modern image generators operate, how prompts and simulation outputs can be translated into visual scenes, and how practitioners can integrate these tools into reproducible local pipelines. By focusing on transferable principles rather than specific tools, the tutorial equips M&S practitioners with the knowledge needed to evaluate, adopt, and adapt text-to-image generation in their simulation workflows.

Related papers

• Artificial Intelligence and Modeling & Simulation: An Overview [2.982218441172364]
  Artificial intelligence (AI) and Modeling & Simulation (M&S) are increasingly intertwined.<n>This report provides a structured overview of the intersections of AI and M&S.
  arXiv  Detail & Related papers  (2026-08-01T00:36:44Z)
• Coding the Visual World: From Image to Simulation Using Vision Language Models [2.6034777771586946]
  This work explores the capacity of Vision Language Models (VLMs) to recognize and simulate systems in images.<n>The VLM is given a natural image of a real-world system and is tasked with describing the system and writing code that simulates and generates it.<n>This approach is tested on various complex emergent systems, ranging from physical systems (waves, lights, clouds) to vegetation, cities, materials, and geological formations.
  arXiv  Detail & Related papers  (2026-01-08T19:49:05Z)
• Janus-Pro-R1: Advancing Collaborative Visual Comprehension and Generation via Reinforcement Learning [92.57052246970254]
  We propose to enable the collaborative co-evolution of visual comprehension and generation.<n>We introduce a two-stage training approach: supervised fine-tuning teaches the MLLM with the foundational ability to generate genuine CoT.<n>We unlock the Aha moment in visual generation, advancing MLLMs from text-to-image tasks to unified image generation.
  arXiv  Detail & Related papers  (2025-06-02T09:39:28Z)
• PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions [65.73013453373349]
  PixWizard is an image-to-image visual assistant designed for image generation, manipulation, and translation based on free-from language instructions.<n>We tackle a variety of vision tasks into a unified image-text-to-image generation framework and curate an Omni Pixel-to-Pixel Instruction-Tuning dataset.<n>Our experiments demonstrate that PixWizard not only shows impressive generative and understanding abilities for images with diverse resolutions but also exhibits promising generalization capabilities with unseen tasks and human instructions.
  arXiv  Detail & Related papers  (2024-09-23T17:59:46Z)
• Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design [0.0]
  Cephalo is a series of vision large language models (V-LLMs) designed for materials science applications. It is trained on integrated image and text data from thousands of scientific papers. Generative applications include bio-inspired designs, including pollen-inspired architected materials.
  arXiv  Detail & Related papers  (2024-05-29T13:34:32Z)
• Instruct-Imagen: Image Generation with Multi-modal Instruction [90.04481955523514]
  instruct-imagen is a model that tackles heterogeneous image generation tasks and generalizes across unseen tasks. We introduce *multi-modal instruction* for image generation, a task representation articulating a range of generation intents with precision. Human evaluation on various image generation datasets reveals that instruct-imagen matches or surpasses prior task-specific models in-domain.
  arXiv  Detail & Related papers  (2024-01-03T19:31:58Z)
• VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation [79.02357561313785]
  We introduce Vision-Language Generative Pre-trained Transformer (VL-GPT), a transformer model proficient at concurrently perceiving and generating visual and linguistic data. VL-GPT achieves a unified pre-training approach for both image and text modalities by employing a straightforward auto-regressive objective.
  arXiv  Detail & Related papers  (2023-12-14T18:59:43Z)
• RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model [93.8067369210696]
  Text-to-image generation (TTI) refers to the usage of models that could process text input and generate high fidelity images based on text descriptions. Diffusion models are one prominent type of generative model used for the generation of images through the systematic introduction of noises with repeating steps. In the era of large models, scaling up model size and the integration with large language models have further improved the performance of TTI models.
  arXiv  Detail & Related papers  (2023-09-02T03:27:20Z)
• Generating Images with Multimodal Language Models [78.6660334861137]
  We propose a method to fuse frozen text-only large language models with pre-trained image encoder and decoder models. Our model demonstrates a wide suite of multimodal capabilities: image retrieval, novel image generation, and multimodal dialogue.
  arXiv  Detail & Related papers  (2023-05-26T19:22:03Z)
• Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation [10.39028769374367]
  We present a new framework that takes text-to-image synthesis to the realm of image-to-image translation. Our method harnesses the power of a pre-trained text-to-image diffusion model to generate a new image that complies with the target text.
  arXiv  Detail & Related papers  (2022-11-22T20:39:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.