Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation

• URL: http://arxiv.org/abs/2503.10845v1
• Date: Thu, 13 Mar 2025 19:54:53 GMT
• Title: Panopticon: Advancing Any-Sensor Foundation Models for Earth Observation
• Authors: Leonard Waldmann, Ando Shah, Yi Wang, Nils Lehmann, Adam J. Stewart, Zhitong Xiong, Xiao Xiang Zhu, Stefan Bauer, John Chuang,
• Abstract summary: We propose Panopticon, an any-sensor foundation model built on the DINOv2 framework.<n>By encoding the wavelength and modes of optical and synthetic aperture radar sensors, Panopticon can effectively process any combination of arbitrary channels.<n>Panopticon enables immediate generalization to both existing and future satellite platforms.
• Score: 31.64195131733122
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Earth observation (EO) data features diverse sensing platforms with varying spectral bands, spatial resolutions, and sensing modalities. While most prior work has constrained inputs to fixed sensors, a new class of any-sensor foundation models able to process arbitrary sensors has recently emerged. Contributing to this line of work, we propose Panopticon, an any-sensor foundation model built on the DINOv2 framework. We extend DINOv2 by (1) treating images of the same geolocation across sensors as natural augmentations, (2) subsampling channels to diversify spectral input, and (3) adding a cross attention over channels as a flexible patch embedding mechanism. By encoding the wavelength and modes of optical and synthetic aperture radar sensors, respectively, Panopticon can effectively process any combination of arbitrary channels. In extensive evaluations, we achieve state-of-the-art performance on GEO-Bench, especially on the widely-used Sentinel-1 and Sentinel-2 sensors, while out-competing other any-sensor models, as well as domain adapted fixed-sensor models on unique sensor configurations. Panopticon enables immediate generalization to both existing and future satellite platforms, advancing sensor-agnostic EO.

Related papers

• FDCT: Frequency-Aware Decomposition and Cross-Modal Token-Alignment for Multi-Sensor Target Classification [10.878168590232852]
  We decompose, align, and fuse multiple image sensor data for target classification. We propose a shared discrete token (UDT) space between sensors to reduce the domain and granularity gaps. We achieve superior classification performance compared to single-modality classifiers.
  arXiv  Detail & Related papers  (2025-03-12T22:12:35Z)
• MSSIDD: A Benchmark for Multi-Sensor Denoising [55.41612200877861]
  We introduce a new benchmark, the Multi-Sensor SIDD dataset, which is the first raw-domain dataset designed to evaluate the sensor transferability of denoising models. We propose a sensor consistency training framework that enables denoising models to learn the sensor-invariant features.
  arXiv  Detail & Related papers  (2024-11-18T13:32:59Z)
• STARS: Sensor-agnostic Transformer Architecture for Remote Sensing [2.6938549839852524]
  We present a sensor-agnostic spectral transformer as the basis for spectral foundation models. We introduce a Universal Spectral Representation (USR) that encodes spectra from any spectral instrument into a common representation. We develop a methodology for pre-training such models in a self-supervised manner.
  arXiv  Detail & Related papers  (2024-11-08T17:16:02Z)
• CAFuser: Condition-Aware Multimodal Fusion for Robust Semantic Perception of Driving Scenes [56.52618054240197]
  We propose a novel, condition-aware multimodal fusion approach for robust semantic perception of driving scenes.<n>Our method, CAFuser, uses an RGB camera input to classify environmental conditions and generate a Condition Token.<n>Our model significantly improves robustness and accuracy, especially in adverse-condition scenarios.
  arXiv  Detail & Related papers  (2024-10-14T17:56:20Z)
• SenPa-MAE: Sensor Parameter Aware Masked Autoencoder for Multi-Satellite Self-Supervised Pretraining [1.4528189330418977]
  SenPa-MAE encodes the sensor parameters of an observed multispectral signal into the image embeddings. SenPa-MAE can be pre-trained on imagery of different satellites with non-matching spectral or geometrical sensor characteristics.
  arXiv  Detail & Related papers  (2024-08-20T16:53:30Z)
• Bridging Remote Sensors with Multisensor Geospatial Foundation Models [15.289711240431107]
  msGFM is a multisensor geospatial foundation model that unifies data from four key sensor modalities. For data originating from identical geolocations, our model employs an innovative cross-sensor pretraining approach. msGFM has demonstrated enhanced proficiency in a range of both single-sensor and multisensor downstream tasks.
  arXiv  Detail & Related papers  (2024-04-01T17:30:56Z)
• Automatic Spatial Calibration of Near-Field MIMO Radar With Respect to Optical Depth Sensors [4.328226032204419]
  We propose a novel, joint calibration approach for optical RGB-D sensors and MIMO radars that is designed to operate in the radar's near-field range. Our pipeline consists of a bespoke calibration target, allowing for automatic target detection and localization. We validate our approach using two different depth sensing technologies from the optical domain.
  arXiv  Detail & Related papers  (2024-03-16T17:24:46Z)
• Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a Light-Weight ToF Sensor [58.305341034419136]
  We present the first dense SLAM system with a monocular camera and a light-weight ToF sensor. We propose a multi-modal implicit scene representation that supports rendering both the signals from the RGB camera and light-weight ToF sensor. Experiments demonstrate that our system well exploits the signals of light-weight ToF sensors and achieves competitive results.
  arXiv  Detail & Related papers  (2023-08-28T07:56:13Z)
• Energy-Based Models for Cross-Modal Localization using Convolutional Transformers [52.27061799824835]
  We present a novel framework for localizing a ground vehicle mounted with a range sensor against satellite imagery in the absence of GPS. We propose a method using convolutional transformers that performs accurate metric-level localization in a cross-modal manner. We train our model end-to-end and demonstrate our approach achieving higher accuracy than the state-of-the-art on KITTI, Pandaset, and a custom dataset.
  arXiv  Detail & Related papers  (2023-06-06T21:27:08Z)
• Drone Detection and Tracking in Real-Time by Fusion of Different Sensing Modalities [66.4525391417921]
  We design and evaluate a multi-sensor drone detection system. Our solution integrates a fish-eye camera as well to monitor a wider part of the sky and steer the other cameras towards objects of interest. The thermal camera is shown to be a feasible solution as good as the video camera, even if the camera employed here has a lower resolution.
  arXiv  Detail & Related papers  (2022-07-05T10:00:58Z)
• Automatic Extrinsic Calibration Method for LiDAR and Camera Sensor Setups [68.8204255655161]
  We present a method to calibrate the parameters of any pair of sensors involving LiDARs, monocular or stereo cameras. The proposed approach can handle devices with very different resolutions and poses, as usually found in vehicle setups.
  arXiv  Detail & Related papers  (2021-01-12T12:02:26Z)
• Deep Soft Procrustes for Markerless Volumetric Sensor Alignment [81.13055566952221]
  In this work, we improve markerless data-driven correspondence estimation to achieve more robust multi-sensor spatial alignment. We incorporate geometric constraints in an end-to-end manner into a typical segmentation based model and bridge the intermediate dense classification task with the targeted pose estimation one. Our model is experimentally shown to achieve similar results with marker-based methods and outperform the markerless ones, while also being robust to the pose variations of the calibration structure.
  arXiv  Detail & Related papers  (2020-03-23T10:51:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.