Compositional Generative Model of Unbounded 4D Cities

• URL: http://arxiv.org/abs/2501.08983v2
• Date: Fri, 14 Mar 2025 12:54:19 GMT
• Title: Compositional Generative Model of Unbounded 4D Cities
• Authors: Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei Liu,
• Abstract summary: We propose a compositional generative model specifically tailored for generating 4D cities.<n>CityDreamer4D supports a range of downstream applications, such as instance editing, city stylization, and urban simulation.
• Score: 44.203932215464214
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 3D scene generation has garnered growing attention in recent years and has made significant progress. Generating 4D cities is more challenging than 3D scenes due to the presence of structurally complex, visually diverse objects like buildings and vehicles, and heightened human sensitivity to distortions in urban environments. To tackle these issues, we propose CityDreamer4D, a compositional generative model specifically tailored for generating unbounded 4D cities. Our main insights are 1) 4D city generation should separate dynamic objects (e.g., vehicles) from static scenes (e.g., buildings and roads), and 2) all objects in the 4D scene should be composed of different types of neural fields for buildings, vehicles, and background stuff. Specifically, we propose Traffic Scenario Generator and Unbounded Layout Generator to produce dynamic traffic scenarios and static city layouts using a highly compact BEV representation. Objects in 4D cities are generated by combining stuff-oriented and instance-oriented neural fields for background stuff, buildings, and vehicles. To suit the distinct characteristics of background stuff and instances, the neural fields employ customized generative hash grids and periodic positional embeddings as scene parameterizations. Furthermore, we offer a comprehensive suite of datasets for city generation, including OSM, GoogleEarth, and CityTopia. The OSM dataset provides a variety of real-world city layouts, while the Google Earth and CityTopia datasets deliver large-scale, high-quality city imagery complete with 3D instance annotations. Leveraging its compositional design, CityDreamer4D supports a range of downstream applications, such as instance editing, city stylization, and urban simulation, while delivering state-of-the-art performance in generating realistic 4D cities.

Related papers

• SynCity: Training-Free Generation of 3D Worlds [107.69875149880679]
  We propose SynCity, a training- and optimization-free approach to generating 3D worlds from textual descriptions. We show how 3D and 2D generators can be combined to generate ever-expanding scenes.
  arXiv  Detail & Related papers  (2025-03-20T17:59:40Z)
• UrbanWorld: An Urban World Model for 3D City Generation [21.21375372182025]
  UrbanWorld is a generative urban world model that can automatically create a customized, realistic and interactive 3D urban world with flexible control conditions. We conduct extensive quantitative analysis on five visual metrics, demonstrating that UrbanWorld achieves SOTA generation realism. We verify the interactive nature of these environments by showcasing the agent perception and navigation within the created environments.
  arXiv  Detail & Related papers  (2024-07-16T17:59:29Z)
• CityCraft: A Real Crafter for 3D City Generation [25.7885801163556]
  CityCraft is an innovative framework designed to enhance both the diversity and quality of urban scene generation. Our approach integrates three key stages: initially, a diffusion transformer (DiT) model is deployed to generate diverse and controllable 2D city layouts. Based on the generated layout and city plan, we utilize the asset retrieval module and Blender for precise asset placement and scene construction.
  arXiv  Detail & Related papers  (2024-06-07T14:49:00Z)
• Comp4D: LLM-Guided Compositional 4D Scene Generation [65.5810466788355]
  We present Comp4D, a novel framework for Compositional 4D Generation. Unlike conventional methods that generate a singular 4D representation of the entire scene, Comp4D innovatively constructs each 4D object within the scene separately. Our method employs a compositional score distillation technique guided by the pre-defined trajectories.
  arXiv  Detail & Related papers  (2024-03-25T17:55:52Z)
• Urban Scene Diffusion through Semantic Occupancy Map [49.20779809250597]
  UrbanDiffusion is a 3D diffusion model conditioned on a Bird's-Eye View (BEV) map. Our model learns the data distribution of scene-level structures within a latent space. After training on real-world driving datasets, our model can generate a wide range of diverse urban scenes.
  arXiv  Detail & Related papers  (2024-03-18T11:54:35Z)
• MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond [69.37319723095746]
  We build a large-scale, comprehensive, and high-quality synthetic dataset for city-scale neural rendering researches. We develop a pipeline to easily collect aerial and street city views, accompanied by ground-truth camera poses and a range of additional data modalities. The resulting pilot dataset, MatrixCity, contains 67k aerial images and 452k street images from two city maps of total size $28km2$.
  arXiv  Detail & Related papers  (2023-09-28T16:06:02Z)
• CityDreamer: Compositional Generative Model of Unbounded 3D Cities [44.203932215464214]
  CityDreamer is a compositional generative model designed specifically for unbounded 3D cities. We adopt the bird's eye view scene representation and employ a volumetric render for both instance-oriented and stuff-oriented neural fields. CityDreamer achieves state-of-the-art performance not only in generating realistic 3D cities but also in localized editing within the generated cities.
  arXiv  Detail & Related papers  (2023-09-01T17:57:02Z)
• UrbanScene3D: A Large Scale Urban Scene Dataset and Simulator [13.510431691480727]
  We present a large scale urban scene dataset associated with a handy simulator based on Unreal Engine 4 and AirSim. Unlike previous works that purely based on 2D information or man-made 3D CAD models, UrbanScene3D contains both compact man-made models and detailed real-world models reconstructed by aerial images.
  arXiv  Detail & Related papers  (2021-07-09T07:56:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.