SegFly: A 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale

• URL: http://arxiv.org/abs/2603.17920v1
• Date: Wed, 18 Mar 2026 16:57:22 GMT
• Title: SegFly: A 2D-3D-2D Paradigm for Aerial RGB-Thermal Semantic Segmentation at Scale
• Abstract summary: We propose a scalable geometry-driven 2D-3D-2D paradigm for semantic segmentation of aerial imagery.<n>By lifting less than 3% of RGB images into a semantic 3D point cloud, we produce 97% of RGB labels and 100% of thermal labels.<n>We extend this 2D-3D-2D paradigm to cross-modal image registration, using 3D geometry as an intermediate alignment space.
• Score: 42.70356809606482
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Semantic segmentation for uncrewed aerial vehicles (UAVs) is fundamental for aerial scene understanding, yet existing RGB and RGB-T datasets remain limited in scale, diversity, and annotation efficiency due to the high cost of manual labeling and the difficulties of accurate RGB-T alignment on off-the-shelf UAVs. To address these challenges, we propose a scalable geometry-driven 2D-3D-2D paradigm that leverages multi-view redundancy in high-overlap aerial imagery to automatically propagate labels from a small subset of manually annotated RGB images to both RGB and thermal modalities within a unified framework. By lifting less than 3% of RGB images into a semantic 3D point cloud and reprojecting it into all views, our approach enables dense pseudo ground-truth generation across large image collections, automatically producing 97% of RGB labels and 100% of thermal labels while achieving 91% and 88% annotation accuracy without any 2D manual refinement. We further extend this 2D-3D-2D paradigm to cross-modal image registration, using 3D geometry as an intermediate alignment space to obtain fully automatic, strong pixel-level RGB-T alignment with 87% registration accuracy and no hardware-level synchronization. Applying our framework to existing geo-referenced aerial imagery, we construct SegFly, a large-scale benchmark with over 20,000 high-resolution RGB images and more than 15,000 geometrically aligned RGB-T pairs spanning diverse urban, industrial, and rural environments across multiple altitudes and seasons. On SegFly, we establish the Firefly baseline for RGB and thermal semantic segmentation and show that both conventional architectures and vision foundation models benefit substantially from SegFly supervision, highlighting the potential of geometry-driven 2D-3D-2D pipelines for scalable multi-modal scene understanding. Data and Code available at https://github.com/markus-42/SegFly.

Related papers

• Demo-Pose: Depth-Monocular Modality Fusion For Object Pose Estimation [5.466547563815996]
  We address the challenge of category-level 9-DoF pose estimation from RGB-D input, without relying on CAD models during inference.<n>We propose DeMo-Pose, a hybrid architecture that fuses semantic features with depth-based graph convolutional representations.<n>Our approach achieves real-time inference and significantly improves over state-of-the-art methods across object categories.
  arXiv  Detail & Related papers  (2026-03-29T05:58:04Z)
• Bidirectional Cross-Attention Fusion of High-Res RGB and Low-Res HSI for Multimodal Automated Waste Sorting [0.7166329363666318]
  We present Bi Cross-Attention Fusion (BCAF), which aligns high-resolution RGB with low-resolution HSI at their native grids via localized, bidirectional cross-attention.<n>BCAF achieves state-of-the-art performance of 76.4% mIoU at 31 images/s and 75.4% mIoU at 55 images/s.
  arXiv  Detail & Related papers  (2026-03-14T13:21:36Z)
• Top2Ground: A Height-Aware Dual Conditioning Diffusion Model for Robust Aerial-to-Ground View Generation [14.377332218510743]
  Top2Ground is a novel diffusion-based method that directly generates ground-view images from aerial input images.<n>We condition the denoising process on a joint representation of VAE-encoded spatial features.<n>Top2Ground can robustly handle both wide and narrow fields of view, highlighting its strong generalization capabilities.
  arXiv  Detail & Related papers  (2025-11-11T13:53:07Z)
• Vision-Language Alignment from Compressed Image Representations using 2D Gaussian Splatting [4.2390854432099205]
  Modern vision language pipelines are driven by RGB vision encoders trained on massive image text corpora.<n>These pipelines inherit two structural inefficiencies from the pixel domain: (i) transmitting dense RGB images from edge devices to the cloud is energy intensive and costly, and (ii) patch based tokenization explodes sequence length.<n>We explore 2D Gaussian Splatting (2DGS) as an alternative visual substrate for alignment.
  arXiv  Detail & Related papers  (2025-09-26T17:41:57Z)
• DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation [66.7989548848166]
  Existing approaches encode depth maps along with RGB images and perform feature fusion between them to enable more robust predictions.<n>We present DFormerv2, a strong RGBD encoder that explicitly uses depth maps as geometry priors rather than encoding depth information with neural networks.<n>Our goal is to extract the geometry clues from the depth and spatial distances among all the image patch tokens, which will then be used as geometry priors to allocate attention weights in self-attention.
  arXiv  Detail & Related papers  (2025-04-07T03:06:07Z)
• Towards Human-Level 3D Relative Pose Estimation: Generalizable, Training-Free, with Single Reference [65.42565481489132]
  Humans can easily deduce the relative pose of a previously unseen object, without labeling or training, given only a single query-reference image pair.<n>We propose a novel 3D generalizable relative pose estimation method by elaborating 3D/2.5D shape perception with a 2.5D shape from RGB-D reference.<n>Our differentiable takes the 2.5D rotatable mesh textured by the RGB and the semantic maps (obtained by DINOv2 from the RGB input), then renders new RGB and semantic maps under a novel rotated view.
  arXiv  Detail & Related papers  (2024-06-26T16:01:10Z)
• DFormer: Rethinking RGBD Representation Learning for Semantic Segmentation [76.81628995237058]
  DFormer is a novel framework to learn transferable representations for RGB-D segmentation tasks. It pretrains the backbone using image-depth pairs from ImageNet-1K. DFormer achieves new state-of-the-art performance on two popular RGB-D tasks.
  arXiv  Detail & Related papers  (2023-09-18T11:09:11Z)
• SSR-2D: Semantic 3D Scene Reconstruction from 2D Images [54.46126685716471]
  In this work, we explore a central 3D scene modeling task, namely, semantic scene reconstruction without using any 3D annotations. The key idea of our approach is to design a trainable model that employs both incomplete 3D reconstructions and their corresponding source RGB-D images. Our method achieves the state-of-the-art performance of semantic scene completion on two large-scale benchmark datasets MatterPort3D and ScanNet.
  arXiv  Detail & Related papers  (2023-02-07T17:47:52Z)
• Boosting RGB-D Saliency Detection by Leveraging Unlabeled RGB Images [89.81919625224103]
  Training deep models for RGB-D salient object detection (SOD) often requires a large number of labeled RGB-D images. We present a Dual-Semi RGB-D Salient Object Detection Network (DS-Net) to leverage unlabeled RGB images for boosting RGB-D saliency detection.
  arXiv  Detail & Related papers  (2022-01-01T03:02:27Z)
• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation [59.94819184452694]
  Depth information has proven to be a useful cue in the semantic segmentation of RGBD images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion. In this paper, we propose a unified and efficient Crossmodality Guided to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively.
  arXiv  Detail & Related papers  (2020-07-17T18:35:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.