DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications

• URL: http://arxiv.org/abs/2502.17066v2
• Date: Wed, 16 Jul 2025 17:16:15 GMT
• Title: DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
• Abstract summary: We present DUNIA, an approach to learn pixel-sized embeddings through cross-modal alignment between images and full-waveform LiDAR data.<n>As the model is trained in a contrastive manner, the embeddings can be directly leveraged in the context of a variety of environmental monitoring tasks in a zero-shot setting.
• Score: 39.0842608370641
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Significant efforts have been directed towards adapting self-supervised multimodal learning for Earth observation applications. However, most current methods produce coarse patch-sized embeddings, limiting their effectiveness and integration with other modalities like LiDAR. To close this gap, we present DUNIA, an approach to learn pixel-sized embeddings through cross-modal alignment between images and full-waveform LiDAR data. As the model is trained in a contrastive manner, the embeddings can be directly leveraged in the context of a variety of environmental monitoring tasks in a zero-shot setting. In our experiments, we demonstrate the effectiveness of the embeddings for seven such tasks: canopy height mapping, fractional canopy cover, land cover mapping, tree species identification, plant area index, crop type classification, and per-pixel waveform-based vertical structure mapping. The results show that the embeddings, along with zero-shot classifiers, often outperform specialized supervised models, even in low-data regimes. In the fine-tuning setting, we show strong performances near or better than the state-of-the-art on five out of six tasks.

Related papers

• Hierarchical Consistency Learning for Test-time Adaptation in Camouflage Perception [50.278200968044665]
  Camouflaged object detection (COD) aims to localize targets that exhibit minimal perceptual differences from backgrounds through physical attributes.<n>Existing methods, constrained by the static train-then-freeze paradigm, suffer from domain rigidity and dependency annotation.<n>We propose the hierarchical consistency learning framework, which integrates test-time adaptation for dynamic representation recalibration.
  arXiv  Detail & Related papers  (2026-05-25T09:57:46Z)
• Better Together: Evaluating the Complementarity of Earth Embedding Models [5.793071003655904]
  Earth embedding models transform Earth observation data into embeddings uniquely tied to locations on the Earth's surface.<n>These models are typically evaluated in isolation, comparing the downstream task performance across different Earth embeddings.<n>We propose assessing Earth embeddings by their complementarity: the performance gain of fused embeddings over the best single-model baseline.
  arXiv  Detail & Related papers  (2026-05-18T17:10:13Z)
• Hi-LOAM: Hierarchical Implicit Neural Fields for LiDAR Odometry and Mapping [0.0]
  We propose a multi-scale implicit neural localization and mapping framework using LiDAR sensor, called Hi-LOAM.<n>Hi-LOAM receives LiDAR point cloud as the input data modality, learns and stores hierarchical latent features in multiple levels of hash tables based on an octree structure.<n>For pose estimation procedure, we rely on a correspondence-free, scan-to-implicit matching paradigm to estimate optimal pose and register current scan into the submap.<n>The entire training process is conducted in a self-supervised manner, which waives the model pre-training and manifests its generalizability when applied to diverse environments.
  arXiv  Detail & Related papers  (2026-04-02T07:36:27Z)
• Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method [54.461213497603154]
  Occupancy-centric methods have recently achieved state-of-the-art results by offering consistent conditioning across frames and modalities.<n>Nuplan-Occ is the largest occupancy dataset to date, constructed from the widely used Nuplan benchmark.<n>We develop a unified framework that jointly synthesizes high-quality occupancy, multi-view videos, and LiDAR point clouds.
  arXiv  Detail & Related papers  (2025-10-27T03:52:45Z)
• LC-SLab -- An Object-based Deep Learning Framework for Large-scale Land Cover Classification from Satellite Imagery and Sparse In-situ Labels [25.42215602005236]
  We propose LC-SLab, a framework for exploring object-based deep learning methods for large-scale land cover classification under sparse supervision.<n> LC-SLab supports both input-level aggregation via graph neural networks, and output-level aggregation by postprocessing results.<n>Our results show that object-based methods can match or exceed the accuracy of common pixel-wise models while producing substantially more coherent maps.
  arXiv  Detail & Related papers  (2025-09-19T11:08:24Z)
• AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities [5.767156832161819]
  We propose AnySat, a multimodal model based on joint embedding predictive architecture (JEPA) and scale-adaptive spatial encoders.<n>To demonstrate the advantages of this unified approach, we compile GeoPlex, a collection of 5 multimodal datasets with varying characteristics.<n>We then train a single powerful model on these diverse datasets simultaneously.
  arXiv  Detail & Related papers  (2024-12-18T18:11:53Z)
• An Enhanced Classification Method Based on Adaptive Multi-Scale Fusion for Long-tailed Multispectral Point Clouds [67.96583737413296]
  We propose an enhanced classification method based on adaptive multi-scale fusion for MPCs with long-tailed distributions. In the training set generation stage, a grid-balanced sampling strategy is designed to reliably generate training samples from sparse labeled datasets. In the feature learning stage, a multi-scale feature fusion module is proposed to fuse shallow features of land-covers at different scales.
  arXiv  Detail & Related papers  (2024-12-16T03:21:20Z)
• Boosting Few-Shot Detection with Large Language Models and Layout-to-Image Synthesis [1.1633929083694388]
  We propose a framework for enhancing few-shot detection beyond state-of-the-art generative augmentation approaches. We introduce our novel layout-aware CLIP score for sample ranking, enabling tight coupling between generated layouts and images. With our approach, a YOLOX-S baseline is boosted by more than 140%, 50%, 35% in mAP on the COCO 5-,10-, and 30-shot settings.
  arXiv  Detail & Related papers  (2024-10-09T12:57:45Z)
• Enhancing Few-Shot Image Classification through Learnable Multi-Scale Embedding and Attention Mechanisms [1.1557852082644071]
  In the context of few-shot classification, the goal is to train a classifier using a limited number of samples. Traditional metric-based methods exhibit certain limitations in achieving this objective. Our approach involves utilizing a multi-output embedding network that maps samples into distinct feature spaces.
  arXiv  Detail & Related papers  (2024-09-12T12:34:29Z)
• Adapt CLIP as Aggregation Instructor for Image Dehazing [17.29370328189668]
  Most dehazing methods suffer from limited receptive field and do not explore the rich semantic prior encapsulated in vision-language models. We introduce CLIPHaze, a pioneering hybrid framework that synergizes the efficient global modeling of Mamba with the prior knowledge and zero-shot capabilities of CLIP. Our method employs parallel state space model and window-based self-attention to obtain global contextual dependency and local fine-grained perception.
  arXiv  Detail & Related papers  (2024-08-22T11:51:50Z)
• Revisiting Generative Adversarial Networks for Binary Semantic Segmentation on Imbalanced Datasets [20.538287907723713]
  Anomalous crack region detection is a typical binary semantic segmentation task, which aims to detect pixels representing cracks on pavement surface images automatically by algorithms. Existing deep learning-based methods have achieved outcoming results on specific public pavement datasets, but the performance would deteriorate dramatically on imbalanced datasets. We propose a deep learning framework based on conditional Generative Adversarial Networks (cGANs) for the anomalous crack region detection tasks at the pixel level.
  arXiv  Detail & Related papers  (2024-02-03T19:24:40Z)
• Terrain-Informed Self-Supervised Learning: Enhancing Building Footprint Extraction from LiDAR Data with Limited Annotations [1.3243401820948064]
  Building footprint maps offer promise of precise footprint extraction without extensive post-processing. Deep learning methods face challenges in generalization and label efficiency. We propose terrain-aware self-supervised learning tailored to remote sensing.
  arXiv  Detail & Related papers  (2023-11-02T12:34:23Z)
• Grounded Text-to-Image Synthesis with Attention Refocusing [16.9170825951175]
  We reveal the potential causes in the diffusion model's cross-attention and self-attention layers. We propose two novel losses to refocus attention maps according to a given spatial layout during sampling. We show that our proposed attention refocusing effectively improves the controllability of existing approaches.
  arXiv  Detail & Related papers  (2023-06-08T17:59:59Z)
• Rethinking Range View Representation for LiDAR Segmentation [66.73116059734788]
  "Many-to-one" mapping, semantic incoherence, and shape deformation are possible impediments against effective learning from range view projections. We present RangeFormer, a full-cycle framework comprising novel designs across network architecture, data augmentation, and post-processing. We show that, for the first time, a range view method is able to surpass the point, voxel, and multi-view fusion counterparts in the competing LiDAR semantic and panoptic segmentation benchmarks.
  arXiv  Detail & Related papers  (2023-03-09T16:13:27Z)
• Towards Effective Image Manipulation Detection with Proposal Contrastive Learning [61.5469708038966]
  We propose Proposal Contrastive Learning (PCL) for effective image manipulation detection. Our PCL consists of a two-stream architecture by extracting two types of global features from RGB and noise views respectively. Our PCL can be easily adapted to unlabeled data in practice, which can reduce manual labeling costs and promote more generalizable features.
  arXiv  Detail & Related papers  (2022-10-16T13:30:13Z)
• 6D Camera Relocalization in Visually Ambiguous Extreme Environments [79.68352435957266]
  We propose a novel method to reliably estimate the pose of a camera given a sequence of images acquired in extreme environments such as deep seas or extraterrestrial terrains. Our method achieves comparable performance with state-of-the-art methods on the indoor benchmark (7-Scenes dataset) using only 20% training data.
  arXiv  Detail & Related papers  (2022-07-13T16:40:02Z)
• PANet: Perspective-Aware Network with Dynamic Receptive Fields and Self-Distilling Supervision for Crowd Counting [63.84828478688975]
  We propose a novel perspective-aware approach called PANet to address the perspective problem. Based on the observation that the size of the objects varies greatly in one image due to the perspective effect, we propose the dynamic receptive fields (DRF) framework. The framework is able to adjust the receptive field by the dilated convolution parameters according to the input image, which helps the model to extract more discriminative features for each local region.
  arXiv  Detail & Related papers  (2021-10-31T04:43:05Z)
• A Contrastive Learning Approach to Auroral Identification and Classification [0.8399688944263843]
  We present a novel application of unsupervised learning to the task of auroral image classification. We modify and adapt the Simple framework for Contrastive Learning of Representations (SimCLR) algorithm to learn representations of auroral images. Our approach exceeds an established threshold for operational purposes, demonstrating readiness for deployment and utilization.
  arXiv  Detail & Related papers  (2021-09-28T17:51:25Z)
• Scale Aware Adaptation for Land-Cover Classification in Remote Sensing Imagery [4.793219747021116]
  Land-cover classification using remote sensing imagery is an important Earth observation task. The benchmark datasets available for training deep segmentation models in remote sensing imagery tend to be small. We propose a scale aware adversarial learning framework to perform joint cross-location and cross-scale land-cover classification.
  arXiv  Detail & Related papers  (2020-12-08T05:15:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.