Paving the way toward foundation models for irregular and unaligned Satellite Image Time Series

• URL: http://arxiv.org/abs/2407.08448v1
• Date: Thu, 11 Jul 2024 12:42:10 GMT
• Title: Paving the way toward foundation models for irregular and unaligned Satellite Image Time Series
• Authors: Iris Dumeur, Silvia Valero, Jordi Inglada,
• Abstract summary: We propose an ALIgned Sits (ALISE) to produce aligned latent representations of satellite imagery. Unlike SSL models currently available for SITS, ALISE incorporates a flexible query mechanism to project the SITS into a common and learned temporal projection space. The quality of the produced representation is assessed through three downstream tasks: crop segmentation (PASTIS), land cover segmentation (MultiSenGE) and a novel crop change detection dataset.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although recently several foundation models for satellite remote sensing imagery have been proposed, they fail to address major challenges of real/operational applications. Indeed, embeddings that don't take into account the spectral, spatial and temporal dimensions of the data as well as the irregular or unaligned temporal sampling are of little use for most real world uses.As a consequence, we propose an ALIgned Sits Encoder (ALISE), a novel approach that leverages the spatial, spectral, and temporal dimensions of irregular and unaligned SITS while producing aligned latent representations. Unlike SSL models currently available for SITS, ALISE incorporates a flexible query mechanism to project the SITS into a common and learned temporal projection space. Additionally, thanks to a multi-view framework, we explore integration of instance discrimination along a masked autoencoding task to SITS. The quality of the produced representation is assessed through three downstream tasks: crop segmentation (PASTIS), land cover segmentation (MultiSenGE), and a novel crop change detection dataset. Furthermore, the change detection task is performed without supervision. The results suggest that the use of aligned representations is more effective than previous SSL methods for linear probing segmentation tasks.

Related papers

• Empowering Snapshot Compressive Imaging: Spatial-Spectral State Space Model with Across-Scanning and Local Enhancement [51.557804095896174]
  We introduce a State Space Model with Across-Scanning and Local Enhancement, named ASLE-SSM, that employs a Spatial-Spectral SSM for global-local balanced context encoding and cross-channel interaction promoting. Experimental results illustrate ASLE-SSM's superiority over existing state-of-the-art methods, with an inference speed 2.4 times faster than Transformer-based MST and saving 0.12 (M) of parameters.
  arXiv  Detail & Related papers  (2024-08-01T15:14:10Z)
• PeFAD: A Parameter-Efficient Federated Framework for Time Series Anomaly Detection [51.20479454379662]
  We propose a. Federated Anomaly Detection framework named PeFAD with the increasing privacy concerns. We conduct extensive evaluations on four real datasets, where PeFAD outperforms existing state-of-the-art baselines by up to 28.74%.
  arXiv  Detail & Related papers  (2024-06-04T13:51:08Z)
• GaitASMS: Gait Recognition by Adaptive Structured Spatial Representation and Multi-Scale Temporal Aggregation [2.0444600042188448]
  Gait recognition is one of the most promising video-based biometric technologies. We propose a novel gait recognition framework, denoted as GaitASMS. It can effectively extract the adaptive structured spatial representations and naturally aggregate the multi-scale temporal information.
  arXiv  Detail & Related papers  (2023-07-29T13:03:17Z)
• Revisiting the Encoding of Satellite Image Time Series [2.5874041837241304]
  Image Time Series (SITS)temporal learning is complex due to hightemporal resolutions and irregular acquisition times. We develop a novel perspective of SITS processing as a direct set prediction problem, inspired by the recent trend in adopting query-based transformer decoders. We attain new state-of-the-art (SOTA) results on the Satellite PASTIS benchmark dataset.
  arXiv  Detail & Related papers  (2023-05-03T12:44:20Z)
• Spatial-Aware Token for Weakly Supervised Object Localization [137.0570026552845]
  We propose a task-specific spatial-aware token to condition localization in a weakly supervised manner. Experiments show that the proposed SAT achieves state-of-the-art performance on both CUB-200 and ImageNet, with 98.45% and 73.13% GT-known Loc.
  arXiv  Detail & Related papers  (2023-03-18T15:38:17Z)
• ViTs for SITS: Vision Transformers for Satellite Image Time Series [52.012084080257544]
  We introduce a fully-attentional model for general Satellite Image Time Series (SITS) processing based on the Vision Transformer (ViT) TSViT splits a SITS record into non-overlapping patches in space and time which are tokenized and subsequently processed by a factorized temporo-spatial encoder.
  arXiv  Detail & Related papers  (2023-01-12T11:33:07Z)
• Latent Space Unsupervised Semantic Segmentation [0.0]
  Traditional change-point detection algorithms come with drawbacks, limiting their real-world applicability. This work proposes a novel unsupervised multidimensional time series named Latent Space Unsupervised (LS-USS) LS-USS systematically achieves on par or better performances in both the offline and real-time setting.
  arXiv  Detail & Related papers  (2022-07-22T13:11:42Z)
• SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery [74.82821342249039]
  We present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE) To leverage temporal information, we include a temporal embedding along with independently masking image patches across time.
  arXiv  Detail & Related papers  (2022-07-17T01:35:29Z)
• Investigating Temporal Convolutional Neural Networks for Satellite Image Time Series Classification: A survey [0.0]
  Temporal CNNs have been employed for SITS classification tasks with encouraging results. This paper seeks to survey this method against a plethora of other contemporary methods for SITS classification to validate the existing findings in recent literature. Experiments are carried out on two benchmark SITS datasets with the results demonstrating that Temporal CNNs display a superior performance to the comparative benchmark algorithms.
  arXiv  Detail & Related papers  (2022-04-13T14:08:14Z)
• Tampered VAE for Improved Satellite Image Time Series Classification [1.933681537640272]
  Pyramid Time-Series Transformer (PTST) operates solely on the temporal dimension. We propose a classification-friendly VAE framework that introduces clustering mechanisms into latent space. We hope the proposed framework can serve as a baseline for crop classification with SITS for its modularity and simplicity.
  arXiv  Detail & Related papers  (2022-03-30T08:48:06Z)
• Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction [138.04956118993934]
  We propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST) CST embedding HSI sparsity into deep learning for HSI reconstruction. In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing.
  arXiv  Detail & Related papers  (2022-03-09T16:17:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.