CityCraft: A Real Crafter for 3D City Generation

• URL: http://arxiv.org/abs/2406.04983v1
• Date: Fri, 7 Jun 2024 14:49:00 GMT
• Title: CityCraft: A Real Crafter for 3D City Generation
• Authors: Jie Deng, Wenhao Chai, Junsheng Huang, Zhonghan Zhao, Qixuan Huang, Mingyan Gao, Jianshu Guo, Shengyu Hao, Wenhao Hu, Jenq-Neng Hwang, Xi Li, Gaoang Wang,
• Abstract summary: CityCraft is an innovative framework designed to enhance both the diversity and quality of urban scene generation. Our approach integrates three key stages: initially, a diffusion transformer (DiT) model is deployed to generate diverse and controllable 2D city layouts. Based on the generated layout and city plan, we utilize the asset retrieval module and Blender for precise asset placement and scene construction.
• Score: 25.7885801163556
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: City scene generation has gained significant attention in autonomous driving, smart city development, and traffic simulation. It helps enhance infrastructure planning and monitoring solutions. Existing methods have employed a two-stage process involving city layout generation, typically using Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), or Transformers, followed by neural rendering. These techniques often exhibit limited diversity and noticeable artifacts in the rendered city scenes. The rendered scenes lack variety, resembling the training images, resulting in monotonous styles. Additionally, these methods lack planning capabilities, leading to less realistic generated scenes. In this paper, we introduce CityCraft, an innovative framework designed to enhance both the diversity and quality of urban scene generation. Our approach integrates three key stages: initially, a diffusion transformer (DiT) model is deployed to generate diverse and controllable 2D city layouts. Subsequently, a Large Language Model(LLM) is utilized to strategically make land-use plans within these layouts based on user prompts and language guidelines. Based on the generated layout and city plan, we utilize the asset retrieval module and Blender for precise asset placement and scene construction. Furthermore, we contribute two new datasets to the field: 1)CityCraft-OSM dataset including 2D semantic layouts of urban areas, corresponding satellite images, and detailed annotations. 2) CityCraft-Buildings dataset, featuring thousands of diverse, high-quality 3D building assets. CityCraft achieves state-of-the-art performance in generating realistic 3D cities.

Related papers

• CityDreamer4D: Compositional Generative Model of Unbounded 4D Cities [44.203932215464214]
  We propose a compositional generative model specifically tailored for generating 4D cities. CityDreamer4D supports a range of downstream applications, such as instance editing, city stylization, and urban simulation.
  arXiv  Detail & Related papers  (2025-01-15T17:59:56Z)
• Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians [65.09942210464747]
  Building asset creation is labor-intensive and requires specialized skills to develop design rules. Recent generative models for building creation often overlook these patterns, leading to low visual fidelity and limited scalability. By manipulating procedural code, we can streamline this process and generate an infinite variety of buildings.
  arXiv  Detail & Related papers  (2024-12-10T16:45:32Z)
• CityX: Controllable Procedural Content Generation for Unbounded 3D Cities [50.10101235281943]
  Current generative methods fall short in either diversity, controllability, or fidelity. In this work, we resort to the procedural content generation (PCG) technique for high-fidelity generation. We develop a multi-agent framework to transform multi-modal instructions, including OSM, semantic maps, and satellite images, into executable programs. Our method, named CityX, demonstrates its superiority in creating diverse, controllable, and realistic 3D urban scenes.
  arXiv  Detail & Related papers  (2024-07-24T18:05:13Z)
• Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior [43.14168074750301]
  We introduce a compositional 3D layout representation into text-to-3D paradigm, serving as an additional prior. It comprises a set of semantic primitives with simple geometric structures and explicit arrangement relationships. We also present various scene editing demonstrations, showing the powers of steerable urban scene generation.
  arXiv  Detail & Related papers  (2024-04-10T06:41:30Z)
• Urban Scene Diffusion through Semantic Occupancy Map [49.20779809250597]
  UrbanDiffusion is a 3D diffusion model conditioned on a Bird's-Eye View (BEV) map. Our model learns the data distribution of scene-level structures within a latent space. After training on real-world driving datasets, our model can generate a wide range of diverse urban scenes.
  arXiv  Detail & Related papers  (2024-03-18T11:54:35Z)
• Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion [77.34078223594686]
  We propose a novel architecture for direct 3D scene generation by introducing diffusion models into 3D sparse representations and combining them with neural rendering techniques. Specifically, our approach generates texture colors at the point level for a given geometry using a 3D diffusion model first, which is then transformed into a scene representation in a feed-forward manner. Experiments in two city-scale datasets show that our model demonstrates proficiency in generating photo-realistic street-view image sequences and cross-view urban scenes from satellite imagery.
  arXiv  Detail & Related papers  (2024-01-19T16:15:37Z)
• CityGen: Infinite and Controllable 3D City Layout Generation [26.1563802843242]
  CityGen is a novel end-to-end framework for infinite, diverse and controllable 3D city layout generation. CityGen achieves state-of-the-art (SOTA) performance under FID and KID in generating an infinite and controllable 3D city layout.
  arXiv  Detail & Related papers  (2023-12-03T21:16:37Z)
• CityDreamer: Compositional Generative Model of Unbounded 3D Cities [44.203932215464214]
  CityDreamer is a compositional generative model designed specifically for unbounded 3D cities. We adopt the bird's eye view scene representation and employ a volumetric render for both instance-oriented and stuff-oriented neural fields. CityDreamer achieves state-of-the-art performance not only in generating realistic 3D cities but also in localized editing within the generated cities.
  arXiv  Detail & Related papers  (2023-09-01T17:57:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.