CityX: Controllable Procedural Content Generation for Unbounded 3D Cities

• URL: http://arxiv.org/abs/2407.17572v3
• Date: Tue, 6 Aug 2024 07:36:21 GMT
• Title: CityX: Controllable Procedural Content Generation for Unbounded 3D Cities
• Authors: Shougao Zhang, Mengqi Zhou, Yuxi Wang, Chuanchen Luo, Rongyu Wang, Yiwei Li, Xucheng Yin, Zhaoxiang Zhang, Junran Peng,
• Abstract summary: We propose a novel multi-modal controllable procedural content generation method, named CityX. It enhances realistic, unbounded 3D city generation guided by multiple layout conditions, including OSM, semantic maps, and satellite images. Through this effective framework, CityX shows the potential to build an innovative ecosystem for 3D scene generation.
• Score: 55.737060358043536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating a realistic, large-scale 3D virtual city remains a complex challenge due to the involvement of numerous 3D assets, various city styles, and strict layout constraints. Existing approaches provide promising attempts at procedural content generation to create large-scale scenes using Blender agents. However, they face crucial issues such as difficulties in scaling up generation capability and achieving fine-grained control at the semantic layout level. To address these problems, we propose a novel multi-modal controllable procedural content generation method, named CityX, which enhances realistic, unbounded 3D city generation guided by multiple layout conditions, including OSM, semantic maps, and satellite images. Specifically, the proposed method contains a general protocol for integrating various PCG plugins and a multi-agent framework for transforming instructions into executable Blender actions. Through this effective framework, CityX shows the potential to build an innovative ecosystem for 3D scene generation by bridging the gap between the quality of generated assets and industrial requirements. Extensive experiments have demonstrated the effectiveness of our method in creating high-quality, diverse, and unbounded cities guided by multi-modal conditions. Our project page: https://cityx-lab.github.io.

Related papers

• GenEx: Generating an Explorable World [59.0666303068111]
  We introduce GenEx, a system capable of planning complex embodied world exploration, guided by its generative imagination. GenEx generates an entire 3D-consistent imaginative environment from as little as a single RGB image. GPT-assisted agents are equipped to perform complex embodied tasks, including both goal-agnostic exploration and goal-driven navigation.
  arXiv  Detail & Related papers  (2024-12-12T18:59:57Z)
• Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians [65.09942210464747]
  Building asset creation is labor-intensive and requires specialized skills to develop design rules. Recent generative models for building creation often overlook these patterns, leading to low visual fidelity and limited scalability. By manipulating procedural code, we can streamline this process and generate an infinite variety of buildings.
  arXiv  Detail & Related papers  (2024-12-10T16:45:32Z)
• GaussianAnything: Interactive Point Cloud Latent Diffusion for 3D Generation [75.39457097832113]
  This paper introduces a novel 3D generation framework, offering scalable, high-quality 3D generation with an interactive Point Cloud-structured Latent space. Our framework employs a Variational Autoencoder with multi-view posed RGB-D(epth)-N(ormal) renderings as input, using a unique latent space design that preserves 3D shape information. The proposed method, GaussianAnything, supports multi-modal conditional 3D generation, allowing for point cloud, caption, and single/multi-view image inputs.
  arXiv  Detail & Related papers  (2024-11-12T18:59:32Z)
• LogiCity: Advancing Neuro-Symbolic AI with Abstract Urban Simulation [60.920536939067524]
  We introduce LogiCity, the first simulator based on customizable first-order logic (FOL) for an urban-like environment with multiple dynamic agents. LogiCity models diverse urban elements using semantic and spatial concepts, such as IsAmbulance(X) and IsClose(X, Y) Key feature of LogiCity is its support for user-configurable abstractions, enabling customizable simulation complexities for logical reasoning.
  arXiv  Detail & Related papers  (2024-11-01T17:59:46Z)
• 3D Question Answering for City Scene Understanding [12.433903847890322]
  3D multimodal question answering (MQA) plays a crucial role in scene understanding by enabling intelligent agents to comprehend their surroundings in 3D environments. We introduce a novel 3D MQA dataset named City-3DQA for city-level scene understanding. A new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94 % and 63.76 % in different settings of City-3DQA.
  arXiv  Detail & Related papers  (2024-07-24T16:22:27Z)
• UrbanWorld: An Urban World Model for 3D City Generation [21.21375372182025]
  UrbanWorld is a generative urban world model that can automatically create a customized, realistic and interactive 3D urban world with flexible control conditions. We conduct extensive quantitative analysis on five visual metrics, demonstrating that UrbanWorld achieves SOTA generation realism. We verify the interactive nature of these environments by showcasing the agent perception and navigation within the created environments.
  arXiv  Detail & Related papers  (2024-07-16T17:59:29Z)
• CityCraft: A Real Crafter for 3D City Generation [25.7885801163556]
  CityCraft is an innovative framework designed to enhance both the diversity and quality of urban scene generation. Our approach integrates three key stages: initially, a diffusion transformer (DiT) model is deployed to generate diverse and controllable 2D city layouts. Based on the generated layout and city plan, we utilize the asset retrieval module and Blender for precise asset placement and scene construction.
  arXiv  Detail & Related papers  (2024-06-07T14:49:00Z)
• Coin3D: Controllable and Interactive 3D Assets Generation with Proxy-Guided Conditioning [52.81032340916171]
  Coin3D allows users to control the 3D generation using a coarse geometry proxy assembled from basic shapes. Our method achieves superior controllability and flexibility in the 3D assets generation task.
  arXiv  Detail & Related papers  (2024-05-13T17:56:13Z)
• Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior [43.14168074750301]
  We introduce a compositional 3D layout representation into text-to-3D paradigm, serving as an additional prior. It comprises a set of semantic primitives with simple geometric structures and explicit arrangement relationships. We also present various scene editing demonstrations, showing the powers of steerable urban scene generation.
  arXiv  Detail & Related papers  (2024-04-10T06:41:30Z)
• SceneX:Procedural Controllable Large-scale Scene Generation via Large-language Models [53.961002112433576]
  We introduce a large-scale scene generation framework, SceneX, which can automatically produce high-quality procedural models according to designers' textual descriptions. Our SceneX can generate a city spanning 2.5 km times 2.5 km with delicate geometric layout and structures, drastically reducing the time cost from several weeks for professional PCG engineers to just a few hours for an ordinary user.
  arXiv  Detail & Related papers  (2024-03-23T03:23:29Z)
• GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splatting [52.150502668874495]
  We present GALA3D, generative 3D GAussians with LAyout-guided control, for effective compositional text-to-3D generation. GALA3D is a user-friendly, end-to-end framework for state-of-the-art scene-level 3D content generation and controllable editing.
  arXiv  Detail & Related papers  (2024-02-11T13:40:08Z)
• CityGen: Infinite and Controllable 3D City Layout Generation [26.1563802843242]
  CityGen is a novel end-to-end framework for infinite, diverse and controllable 3D city layout generation. CityGen achieves state-of-the-art (SOTA) performance under FID and KID in generating an infinite and controllable 3D city layout.
  arXiv  Detail & Related papers  (2023-12-03T21:16:37Z)
• Octopus: Embodied Vision-Language Programmer from Environmental Feedback [58.04529328728999]
  Embodied vision-language models (VLMs) have achieved substantial progress in multimodal perception and reasoning. To bridge this gap, we introduce Octopus, an embodied vision-language programmer that uses executable code generation as a medium to connect planning and manipulation. Octopus is designed to 1) proficiently comprehend an agent's visual and textual task objectives, 2) formulate intricate action sequences, and 3) generate executable code.
  arXiv  Detail & Related papers  (2023-10-12T17:59:58Z)
• CityDreamer: Compositional Generative Model of Unbounded 3D Cities [44.203932215464214]
  CityDreamer is a compositional generative model designed specifically for unbounded 3D cities. We adopt the bird's eye view scene representation and employ a volumetric render for both instance-oriented and stuff-oriented neural fields. CityDreamer achieves state-of-the-art performance not only in generating realistic 3D cities but also in localized editing within the generated cities.
  arXiv  Detail & Related papers  (2023-09-01T17:57:02Z)
• Pushing the Limits of 3D Shape Generation at Scale [65.24420181727615]
  We present a significant breakthrough in 3D shape generation by scaling it to unprecedented dimensions. We have developed a model with an astounding 3.6 billion trainable parameters, establishing it as the largest 3D shape generation model to date, named Argus-3D.
  arXiv  Detail & Related papers  (2023-06-20T13:01:19Z)
• Evaluating Continual Learning Algorithms by Generating 3D Virtual Environments [66.83839051693695]
  Continual learning refers to the ability of humans and animals to incrementally learn over time in a given environment. We propose to leverage recent advances in 3D virtual environments in order to approach the automatic generation of potentially life-long dynamic scenes with photo-realistic appearance. A novel element of this paper is that scenes are described in a parametric way, thus allowing the user to fully control the visual complexity of the input stream the agent perceives.
  arXiv  Detail & Related papers  (2021-09-16T10:37:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.