EarthCrafter: Scalable 3D Earth Generation via Dual-Sparse Latent Diffusion

• URL: http://arxiv.org/abs/2507.16535v2
• Date: Wed, 23 Jul 2025 01:59:09 GMT
• Title: EarthCrafter: Scalable 3D Earth Generation via Dual-Sparse Latent Diffusion
• Authors: Shang Liu, Chenjie Cao, Chaohui Yu, Wen Qian, Jing Wang, Fan Wang,
• Abstract summary: We introduce Aerial-Earth3D, the largest 3D aerial dataset to date, consisting of 50k curated scenes (each measuring 600m x 600m) captured across the U.S. mainland.<n>Each scene provides pose-annotated multi-view images, depth maps, normals, semantic segmentation, and camera poses, with explicit quality control to ensure terrain diversity.<n>We propose EarthCrafter, a tailored framework for large-scale 3D Earth generation via sparse-decoupled latent diffusion.
• Score: 23.3834795181211
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the remarkable developments achieved by recent 3D generation works, scaling these methods to geographic extents, such as modeling thousands of square kilometers of Earth's surface, remains an open challenge. We address this through a dual innovation in data infrastructure and model architecture. First, we introduce Aerial-Earth3D, the largest 3D aerial dataset to date, consisting of 50k curated scenes (each measuring 600m x 600m) captured across the U.S. mainland, comprising 45M multi-view Google Earth frames. Each scene provides pose-annotated multi-view images, depth maps, normals, semantic segmentation, and camera poses, with explicit quality control to ensure terrain diversity. Building on this foundation, we propose EarthCrafter, a tailored framework for large-scale 3D Earth generation via sparse-decoupled latent diffusion. Our architecture separates structural and textural generation: 1) Dual sparse 3D-VAEs compress high-resolution geometric voxels and textural 2D Gaussian Splats (2DGS) into compact latent spaces, largely alleviating the costly computation suffering from vast geographic scales while preserving critical information. 2) We propose condition-aware flow matching models trained on mixed inputs (semantics, images, or neither) to flexibly model latent geometry and texture features independently. Extensive experiments demonstrate that EarthCrafter performs substantially better in extremely large-scale generation. The framework further supports versatile applications, from semantic-guided urban layout generation to unconditional terrain synthesis, while maintaining geographic plausibility through our rich data priors from Aerial-Earth3D. Our project page is available at https://whiteinblue.github.io/earthcrafter/

Related papers

• Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting [64.64738535860351]
  We present a scalable pipeline that converts single-view images into comprehensive, scale- and appearance-realistic 3D representations.<n>Our method bridges the gap between the vast repository of imagery and the increasing demand for spatial scene understanding.<n>By automatically generating authentic, scale-aware 3D data from images, we significantly reduce data collection costs and open new avenues for advancing spatial intelligence.
  arXiv  Detail & Related papers  (2025-07-24T14:53:26Z)
• GeoProg3D: Compositional Visual Reasoning for City-Scale 3D Language Fields [25.969442927216893]
  GeoProg3D is a visual programming framework that enables natural language-driven interactions with city-scale high-fidelity 3D scenes.<n>Our framework employs large language models (LLMs) as reasoning engines to dynamically combine GV-APIs and operate GCLF.<n>Experiments demonstrate that GeoProg3D significantly outperforms existing 3D language fields and vision-language models across multiple tasks.
  arXiv  Detail & Related papers  (2025-06-29T18:03:03Z)
• REArtGS: Reconstructing and Generating Articulated Objects via 3D Gaussian Splatting with Geometric and Motion Constraints [48.80178020541189]
  REArtGS is a novel framework that introduces additional geometric and motion constraints to 3D Gaussian primitives.<n>We establish deformable fields for 3D Gaussians constrained by the kinematic structures of articulated objects, achieving unsupervised generation of surface meshes in unseen states.
  arXiv  Detail & Related papers  (2025-03-09T16:05:36Z)
• CULTURE3D: A Large-Scale and Diverse Dataset of Cultural Landmarks and Terrains for Gaussian-Based Scene Rendering [12.299096433876676]
  Current state-of-the-art 3D reconstruction models face limitations in building extra-large scale outdoor scenes.<n>In this paper, we present a extra-large fine-grained dataset with 10 billion points composed of 41,006 drone-captured high-resolution aerial images.<n>Compared to existing datasets, ours offers significantly larger scale and higher detail, uniquely suited for fine-grained 3D applications.
  arXiv  Detail & Related papers  (2025-01-12T20:36:39Z)
• Skyeyes: Ground Roaming using Aerial View Images [9.159470619808127]
  We introduce Skyeyes, a novel framework that can generate sequences of ground view images using only aerial view inputs. More specifically, we combine a 3D representation with a view consistent generation model, which ensures coherence between generated images. The images maintain improved spatial-temporal coherence and realism, enhancing scene comprehension and visualization from aerial perspectives.
  arXiv  Detail & Related papers  (2024-09-25T07:21:43Z)
• EarthGen: Generating the World from Top-Down Views [23.66194982885544]
  We present a novel method for extensive multi-scale generative terrain modeling. At the core of our model is a cascade of superresolution diffusion models that can be combined to produce consistent images across multiple resolutions. We evaluate our method on a dataset collected from Bing Maps and show that it outperforms super-resolution baselines on the extreme super-resolution task of 1024x zoom.
  arXiv  Detail & Related papers  (2024-09-02T23:17:56Z)
• GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation [65.33726478659304]
  We introduce the Geometry-Aware Large Reconstruction Model (GeoLRM), an approach which can predict high-quality assets with 512k Gaussians and 21 input images in only 11 GB GPU memory. Previous works neglect the inherent sparsity of 3D structure and do not utilize explicit geometric relationships between 3D and 2D images. GeoLRM tackles these issues by incorporating a novel 3D-aware transformer structure that directly processes 3D points and uses deformable cross-attention mechanisms.
  arXiv  Detail & Related papers  (2024-06-21T17:49:31Z)
• Outdoor Scene Extrapolation with Hierarchical Generative Cellular Automata [70.9375320609781]
  We aim to generate fine-grained 3D geometry from large-scale sparse LiDAR scans, abundantly captured by autonomous vehicles (AV) We propose hierarchical Generative Cellular Automata (hGCA), a spatially scalable 3D generative model, which grows geometry with local kernels following, in a coarse-to-fine manner, equipped with a light-weight planner to induce global consistency.
  arXiv  Detail & Related papers  (2024-06-12T14:56:56Z)
• GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image [94.56927147492738]
  We introduce GeoWizard, a new generative foundation model designed for estimating geometric attributes from single images. We show that leveraging diffusion priors can markedly improve generalization, detail preservation, and efficiency in resource usage. We propose a simple yet effective strategy to segregate the complex data distribution of various scenes into distinct sub-distributions.
  arXiv  Detail & Related papers  (2024-03-18T17:50:41Z)
• Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion [77.34078223594686]
  We propose a novel architecture for direct 3D scene generation by introducing diffusion models into 3D sparse representations and combining them with neural rendering techniques. Specifically, our approach generates texture colors at the point level for a given geometry using a 3D diffusion model first, which is then transformed into a scene representation in a feed-forward manner. Experiments in two city-scale datasets show that our model demonstrates proficiency in generating photo-realistic street-view image sequences and cross-view urban scenes from satellite imagery.
  arXiv  Detail & Related papers  (2024-01-19T16:15:37Z)
• 3D Crowd Counting via Geometric Attention-guided Multi-View Fusion [50.520192402702015]
  We propose to solve the multi-view crowd counting task through 3D feature fusion with 3D scene-level density maps. Compared to 2D fusion, the 3D fusion extracts more information of the people along the z-dimension (height), which helps to address the scale variations across multiple views. The 3D density maps still preserve the 2D density maps property that the sum is the count, while also providing 3D information about the crowd density.
  arXiv  Detail & Related papers  (2020-03-18T11:35:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.