CityX: Controllable Procedural Content Generation for Unbounded 3D Cities

• URL: http://arxiv.org/abs/2407.17572v4
• Date: Mon, 09 Dec 2024 09:30:26 GMT
• Title: CityX: Controllable Procedural Content Generation for Unbounded 3D Cities
• Authors: Shougao Zhang, Mengqi Zhou, Yuxi Wang, Chuanchen Luo, Rongyu Wang, Yiwei Li, Zhaoxiang Zhang, Junran Peng,
• Abstract summary: Current generative methods fall short in either diversity, controllability, or fidelity. In this work, we resort to the procedural content generation (PCG) technique for high-fidelity generation. We develop a multi-agent framework to transform multi-modal instructions, including OSM, semantic maps, and satellite images, into executable programs. Our method, named CityX, demonstrates its superiority in creating diverse, controllable, and realistic 3D urban scenes.
• Score: 50.10101235281943
• License:
• Abstract: Urban areas, as the primary human habitat in modern civilization, accommodate a broad spectrum of social activities. With the surge of embodied intelligence, recent years have witnessed an increasing presence of physical agents in urban areas, such as autonomous vehicles and delivery robots. As a result, practitioners significantly value crafting authentic, simulation-ready 3D cities to facilitate the training and verification of such agents. However, this task is quite challenging. Current generative methods fall short in either diversity, controllability, or fidelity. In this work, we resort to the procedural content generation (PCG) technique for high-fidelity generation. It assembles superior assets according to empirical rules, ultimately leading to industrial-grade outcomes. To ensure diverse and self contained creation, we design a management protocol to accommodate extensive PCG plugins with distinct functions and interfaces. Based on this unified PCG library, we develop a multi-agent framework to transform multi-modal instructions, including OSM, semantic maps, and satellite images, into executable programs. The programs coordinate relevant plugins to construct the 3D city consistent with the control condition. A visual feedback scheme is introduced to further refine the initial outcomes. Our method, named CityX, demonstrates its superiority in creating diverse, controllable, and realistic 3D urban scenes. The synthetic scenes can be seamlessly deployed as a real-time simulator and an infinite data generator for embodied intelligence research. Our project page: https://cityx-lab.github.io.

Related papers

• GenEx: Generating an Explorable World [59.0666303068111]
  We introduce GenEx, a system capable of planning complex embodied world exploration, guided by its generative imagination. GenEx generates an entire 3D-consistent imaginative environment from as little as a single RGB image. GPT-assisted agents are equipped to perform complex embodied tasks, including both goal-agnostic exploration and goal-driven navigation.
  arXiv  Detail & Related papers  (2024-12-12T18:59:57Z)
• Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians [65.09942210464747]
  Building asset creation is labor-intensive and requires specialized skills to develop design rules. Recent generative models for building creation often overlook these patterns, leading to low visual fidelity and limited scalability. By manipulating procedural code, we can streamline this process and generate an infinite variety of buildings.
  arXiv  Detail & Related papers  (2024-12-10T16:45:32Z)
• LogiCity: Advancing Neuro-Symbolic AI with Abstract Urban Simulation [60.920536939067524]
  We introduce LogiCity, the first simulator based on customizable first-order logic (FOL) for an urban-like environment with multiple dynamic agents. LogiCity models diverse urban elements using semantic and spatial concepts, such as IsAmbulance(X) and IsClose(X, Y) Key feature of LogiCity is its support for user-configurable abstractions, enabling customizable simulation complexities for logical reasoning.
  arXiv  Detail & Related papers  (2024-11-01T17:59:46Z)
• 3D Question Answering for City Scene Understanding [12.433903847890322]
  3D multimodal question answering (MQA) plays a crucial role in scene understanding by enabling intelligent agents to comprehend their surroundings in 3D environments. We introduce a novel 3D MQA dataset named City-3DQA for city-level scene understanding. A new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94 % and 63.76 % in different settings of City-3DQA.
  arXiv  Detail & Related papers  (2024-07-24T16:22:27Z)
• UrbanWorld: An Urban World Model for 3D City Generation [21.21375372182025]
  UrbanWorld is a generative urban world model that can automatically create a customized, realistic and interactive 3D urban world with flexible control conditions. We conduct extensive quantitative analysis on five visual metrics, demonstrating that UrbanWorld achieves SOTA generation realism. We verify the interactive nature of these environments by showcasing the agent perception and navigation within the created environments.
  arXiv  Detail & Related papers  (2024-07-16T17:59:29Z)
• CityCraft: A Real Crafter for 3D City Generation [25.7885801163556]
  CityCraft is an innovative framework designed to enhance both the diversity and quality of urban scene generation. Our approach integrates three key stages: initially, a diffusion transformer (DiT) model is deployed to generate diverse and controllable 2D city layouts. Based on the generated layout and city plan, we utilize the asset retrieval module and Blender for precise asset placement and scene construction.
  arXiv  Detail & Related papers  (2024-06-07T14:49:00Z)
• CityGen: Infinite and Controllable 3D City Layout Generation [26.1563802843242]
  CityGen is a novel end-to-end framework for infinite, diverse and controllable 3D city layout generation. CityGen achieves state-of-the-art (SOTA) performance under FID and KID in generating an infinite and controllable 3D city layout.
  arXiv  Detail & Related papers  (2023-12-03T21:16:37Z)
• Octopus: Embodied Vision-Language Programmer from Environmental Feedback [58.04529328728999]
  Embodied vision-language models (VLMs) have achieved substantial progress in multimodal perception and reasoning. To bridge this gap, we introduce Octopus, an embodied vision-language programmer that uses executable code generation as a medium to connect planning and manipulation. Octopus is designed to 1) proficiently comprehend an agent's visual and textual task objectives, 2) formulate intricate action sequences, and 3) generate executable code.
  arXiv  Detail & Related papers  (2023-10-12T17:59:58Z)
• Evaluating Continual Learning Algorithms by Generating 3D Virtual Environments [66.83839051693695]
  Continual learning refers to the ability of humans and animals to incrementally learn over time in a given environment. We propose to leverage recent advances in 3D virtual environments in order to approach the automatic generation of potentially life-long dynamic scenes with photo-realistic appearance. A novel element of this paper is that scenes are described in a parametric way, thus allowing the user to fully control the visual complexity of the input stream the agent perceives.
  arXiv  Detail & Related papers  (2021-09-16T10:37:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.