SkyScenes: A Synthetic Dataset for Aerial Scene Understanding

• URL: http://arxiv.org/abs/2312.06719v2
• Date: Sun, 22 Sep 2024 00:23:45 GMT
• Title: SkyScenes: A Synthetic Dataset for Aerial Scene Understanding
• Abstract summary: We present SkyScenes, a dataset of densely annotated aerial images from Unmanned Aerial Vehicle perspectives. We show that models trained on SkyScenes generalize well to different real-world scenarios.
• Score: 13.55299647715449
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-world aerial scene understanding is limited by a lack of datasets that contain densely annotated images curated under a diverse set of conditions. Due to inherent challenges in obtaining such images in controlled real-world settings, we present SkyScenes, a synthetic dataset of densely annotated aerial images captured from Unmanned Aerial Vehicle (UAV) perspectives. We carefully curate SkyScenes images from CARLA to comprehensively capture diversity across layouts (urban and rural maps), weather conditions, times of day, pitch angles and altitudes with corresponding semantic, instance and depth annotations. Through our experiments using SkyScenes, we show that (1) models trained on SkyScenes generalize well to different real-world scenarios, (2) augmenting training on real images with SkyScenes data can improve real-world performance, (3) controlled variations in SkyScenes can offer insights into how models respond to changes in viewpoint conditions (height and pitch), weather and time of day, and (4) incorporating additional sensor modalities (depth) can improve aerial scene understanding. Our dataset and associated generation code are publicly available at: https://hoffman-group.github.io/SkyScenes/

Related papers

• Sky2Ground: A Benchmark for Site Modeling under Varying Altitude [3.479712135769016]
  We introduce Sky2Ground, a three-view dataset designed for varying altitude camera localization, correspondence learning, and reconstruction.<n>The dataset combines structured synthetic imagery with real, in-the-wild images, providing both controlled multi-view geometry and realistic scene noise.<n>We benchmark state of the art pose estimation models, including MASt3R, DUSt3R, Map Anything, and VGGT, and observe that the use of satellite imagery often degrades performance.<n>We propose SkyNet, a model which enhances cross-view consistency when incorporating satellite imagery with a curriculum-based training strategy.
  arXiv  Detail & Related papers  (2026-03-14T03:59:19Z)
• Full Dynamic Range Sky-Modelling For Image Based Lighting [0.0]
  Environment maps are a key component to modelling real-world outdoor scenes.<n>Recent work has demonstrated skymodels to be more comprehensive in accuracy and lighting directionality.
  arXiv  Detail & Related papers  (2026-03-05T23:32:18Z)
• Top2Ground: A Height-Aware Dual Conditioning Diffusion Model for Robust Aerial-to-Ground View Generation [14.377332218510743]
  Top2Ground is a novel diffusion-based method that directly generates ground-view images from aerial input images.<n>We condition the denoising process on a joint representation of VAE-encoded spatial features.<n>Top2Ground can robustly handle both wide and narrow fields of view, highlighting its strong generalization capabilities.
  arXiv  Detail & Related papers  (2025-11-11T13:53:07Z)
• AerialMegaDepth: Learning Aerial-Ground Reconstruction and View Synthesis [57.249817395828174]
  We propose a scalable framework combining pseudo-synthetic renderings from 3D city-wide meshes with real, ground-level crowd-sourced images. The pseudo-synthetic data simulates a wide range of aerial viewpoints, while the real, crowd-sourced images help improve visual fidelity for ground-level images. Using this hybrid dataset, we fine-tune several state-of-the-art algorithms and achieve significant improvements on real-world, zero-shot aerial-ground tasks.
  arXiv  Detail & Related papers  (2025-04-17T17:57:05Z)
• Horizon-GS: Unified 3D Gaussian Splatting for Large-Scale Aerial-to-Ground Scenes [55.15494682493422]
  We introduce Horizon-GS, a novel approach built upon Gaussian Splatting techniques, to tackle the unified reconstruction and rendering for aerial and street views. Our method addresses the key challenges of combining these perspectives with a new training strategy, overcoming viewpoint discrepancies to generate high-fidelity scenes.
  arXiv  Detail & Related papers  (2024-12-02T17:42:00Z)
• SkyAI Sim: An Open-Source Simulation of UAV Aerial Imaging from Satellite Data [0.8749675983608172]
  Capturing real-world aerial images for vision-based navigation (VBN) is challenging due to limited availability and conditions. SkyAI Sim offers a compelling alternative by simulating a UAV to capture bird's-eye view satellite images at zero-yaw with real-world visible-band specifications.
  arXiv  Detail & Related papers  (2024-10-02T20:08:29Z)
• Skyeyes: Ground Roaming using Aerial View Images [9.159470619808127]
  We introduce Skyeyes, a novel framework that can generate sequences of ground view images using only aerial view inputs. More specifically, we combine a 3D representation with a view consistent generation model, which ensures coherence between generated images. The images maintain improved spatial-temporal coherence and realism, enhancing scene comprehension and visualization from aerial perspectives.
  arXiv  Detail & Related papers  (2024-09-25T07:21:43Z)
• Leveraging BEV Paradigm for Ground-to-Aerial Image Synthesis [14.492759165786364]
  Ground-to-aerial image synthesis focuses on generating realistic aerial images from corresponding ground street view images. We introduce SkyDiffusion, a novel cross-view generation method for synthesizing aerial images from street view images. We introduce a novel dataset, Ground2Aerial-3, designed for diverse ground-to-aerial image synthesis applications.
  arXiv  Detail & Related papers  (2024-08-03T15:43:56Z)
• WayveScenes101: A Dataset and Benchmark for Novel View Synthesis in Autonomous Driving [4.911903454560829]
  WayveScenes101 is a dataset designed to help the community advance the state of the art in novel view synthesis. The dataset comprises 101 driving scenes across a wide range of environmental conditions and driving scenarios.
  arXiv  Detail & Related papers  (2024-07-11T08:29:45Z)
• Sat2Scene: 3D Urban Scene Generation from Satellite Images with Diffusion [77.34078223594686]
  We propose a novel architecture for direct 3D scene generation by introducing diffusion models into 3D sparse representations and combining them with neural rendering techniques. Specifically, our approach generates texture colors at the point level for a given geometry using a 3D diffusion model first, which is then transformed into a scene representation in a feed-forward manner. Experiments in two city-scale datasets show that our model demonstrates proficiency in generating photo-realistic street-view image sequences and cross-view urban scenes from satellite imagery.
  arXiv  Detail & Related papers  (2024-01-19T16:15:37Z)
• Multiview Aerial Visual Recognition (MAVREC): Can Multi-view Improve Aerial Visual Perception? [57.77643186237265]
  We present Multiview Aerial Visual RECognition or MAVREC, a video dataset where we record synchronized scenes from different perspectives. MAVREC consists of around 2.5 hours of industry-standard 2.7K resolution video sequences, more than 0.5 million frames, and 1.1 million annotated bounding boxes. This makes MAVREC the largest ground and aerial-view dataset, and the fourth largest among all drone-based datasets.
  arXiv  Detail & Related papers  (2023-12-07T18:59:14Z)
• HawkI: Homography & Mutual Information Guidance for 3D-free Single Image to Aerial View [67.8213192993001]
  We present HawkI, for synthesizing aerial-view images from text and an exemplar image. HawkI blends the visual features from the input image within a pretrained text-to-2Dimage stable diffusion model. At inference, HawkI employs a unique mutual information guidance formulation to steer the generated image towards faithfully replicating the semantic details of the input-image.
  arXiv  Detail & Related papers  (2023-11-27T01:41:25Z)
• CommonScenes: Generating Commonsense 3D Indoor Scenes with Scene Graph Diffusion [83.30168660888913]
  We present CommonScenes, a fully generative model that converts scene graphs into corresponding controllable 3D scenes. Our pipeline consists of two branches, one predicting the overall scene layout via a variational auto-encoder and the other generating compatible shapes. The generated scenes can be manipulated by editing the input scene graph and sampling the noise in the diffusion model.
  arXiv  Detail & Related papers  (2023-05-25T17:39:13Z)
• Danish Airs and Grounds: A Dataset for Aerial-to-Street-Level Place Recognition and Localization [9.834635805575584]
  We contribute with the emphDanish Airs and Grounds dataset, a large collection of street-level and aerial images targeting such cases. The dataset is larger and more diverse than current publicly available data, including more than 50 km of road in urban, suburban and rural areas. We propose a map-to-image re-localization pipeline, that first estimates a dense 3D reconstruction from the aerial images and then matches query street-level images to street-level renderings of the 3D model.
  arXiv  Detail & Related papers  (2022-02-03T19:58:09Z)
• Urban Radiance Fields [77.43604458481637]
  We perform 3D reconstruction and novel view synthesis from data captured by scanning platforms commonly deployed for world mapping in urban outdoor environments. Our approach extends Neural Radiance Fields, which has been demonstrated to synthesize realistic novel images for small scenes in controlled settings. Each of these three extensions provides significant performance improvements in experiments on Street View data.
  arXiv  Detail & Related papers  (2021-11-29T15:58:16Z)
• Neural Sparse Voxel Fields [151.20366604586403]
  We introduce Neural Sparse Voxel Fields (NSVF), a new neural scene representation for fast and high-quality free-viewpoint rendering. NSVF defines a set of voxel-bounded implicit fields organized in a sparse voxel octree to model local properties in each cell. Our method is typically over 10 times faster than the state-of-the-art (namely, NeRF(Mildenhall et al., 2020)) at inference time while achieving higher quality results.
  arXiv  Detail & Related papers  (2020-07-22T17:51:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.