U-TILISE: A Sequence-to-sequence Model for Cloud Removal in Optical Satellite Time Series

• URL: http://arxiv.org/abs/2305.13277v3
• Date: Mon, 4 Dec 2023 21:29:42 GMT
• Title: U-TILISE: A Sequence-to-sequence Model for Cloud Removal in Optical Satellite Time Series
• Authors: Corinne Stucker, Vivien Sainte Fare Garnot, Konrad Schindler
• Abstract summary: We develop a neural model that can map a cloud-masked input sequence to a cloud-free output sequence. We experimentally evaluate the proposed model on a dataset of satellite Sentinel-2 time series acquired all over Europe. Compared to a standard baseline, it increases the PSNR by 1.8 dB at previously seen locations and by 1.3 dB at unseen locations.
• Score: 22.39321609253005
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Satellite image time series in the optical and infrared spectrum suffer from frequent data gaps due to cloud cover, cloud shadows, and temporary sensor outages. It has been a long-standing problem of remote sensing research how to best reconstruct the missing pixel values and obtain complete, cloud-free image sequences. We approach that problem from the perspective of representation learning and develop U-TILISE, an efficient neural model that is able to implicitly capture spatio-temporal patterns of the spectral intensities, and that can therefore be trained to map a cloud-masked input sequence to a cloud-free output sequence. The model consists of a convolutional spatial encoder that maps each individual frame of the input sequence to a latent encoding; an attention-based temporal encoder that captures dependencies between those per-frame encodings and lets them exchange information along the time dimension; and a convolutional spatial decoder that decodes the latent embeddings back into multi-spectral images. We experimentally evaluate the proposed model on EarthNet2021, a dataset of Sentinel-2 time series acquired all over Europe, and demonstrate its superior ability to reconstruct the missing pixels. Compared to a standard interpolation baseline, it increases the PSNR by 1.8 dB at previously seen locations and by 1.3 dB at unseen locations.

Related papers

• bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction [57.199618102578576]
  We propose bit2bit, a new method for reconstructing high-quality image stacks at original resolution from sparse binary quantatemporal image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates a dense image sequence from sparse binary photon data. We present a novel dataset containing a wide range of real SPAD high-speed videos under various challenging imaging conditions.
  arXiv  Detail & Related papers  (2024-10-30T17:30:35Z)
• Cloud gap-filling with deep learning for improved grassland monitoring [2.9272689981427407]
  Uninterrupted optical image time series are crucial for the timely monitoring of agricultural land changes. We propose a deep learning method that integrates cloud-free optical (Sentinel-2) observations and weather-independent (Sentinel-1) Synthetic Aperture Radar (SAR) data.
  arXiv  Detail & Related papers  (2024-03-14T16:41:26Z)
• Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning [59.26623999209235]
  We present DiST, which disentangles the learning of spatial and temporal aspects of videos. The disentangled learning in DiST is highly efficient because it avoids the back-propagation of massive pre-trained parameters. Extensive experiments on five benchmarks show that DiST delivers better performance than existing state-of-the-art methods by convincing gaps.
  arXiv  Detail & Related papers  (2023-09-14T17:58:33Z)
• A Spatial-Temporal Deformable Attention based Framework for Breast Lesion Detection in Videos [107.96514633713034]
  We propose a spatial-temporal deformable attention based framework, named STNet. Our STNet introduces a spatial-temporal deformable attention module to perform local spatial-temporal feature fusion. Experiments on the public breast lesion ultrasound video dataset show that our STNet obtains a state-of-the-art detection performance.
  arXiv  Detail & Related papers  (2023-09-09T07:00:10Z)
• Boosting Point Clouds Rendering via Radiance Mapping [49.24193509772339]
  We focus on boosting the image quality of point clouds rendering with a compact model design. We simplify the NeRF representation to a spatial mapping function which only requires single evaluation per pixel. Our method achieves the state-of-the-art rendering on point clouds, outperforming prior works by notable margins.
  arXiv  Detail & Related papers  (2022-10-27T01:25:57Z)
• Efficient data-driven gap filling of satellite image time series using deep neural networks with partial convolutions [0.0]
  This paper shows how three-dimensional partial convolutions can be used as layers in neural networks to fill gaps in satellite image time series. To evaluate the approach we apply a U-Net-like model on incomplete time series of quasi-global carbon monoxide observations from the Sentinel-5P satellite.
  arXiv  Detail & Related papers  (2022-08-18T11:32:04Z)
• SEN12MS-CR-TS: A Remote Sensing Data Set for Multi-modal Multi-temporal Cloud Removal [15.459106705735376]
  About half of all optical observations collected via spaceborne satellites are affected by haze or clouds. This work addresses the challenge of optical satellite image reconstruction and cloud removal by proposing SEN12MS-CR-TS. We propose two models highlighting the benefits and use cases of SEN12MS-CR-TS.
  arXiv  Detail & Related papers  (2022-01-24T11:38:49Z)
• Dual-view Snapshot Compressive Imaging via Optical Flow Aided Recurrent Neural Network [14.796204921975733]
  Dual-view snapshot compressive imaging (SCI) aims to capture videos from two field-of-views (FoVs) in a single snapshot. It is challenging for existing model-based decoding algorithms to reconstruct each individual scene. We propose an optical flow-aided recurrent neural network for dual video SCI systems, which provides high-quality decoding in seconds.
  arXiv  Detail & Related papers  (2021-09-11T14:24:44Z)
• Sentinel-1 and Sentinel-2 Spatio-Temporal Data Fusion for Clouds Removal [51.9654625216266]
  A novel method for clouds-corrupted optical image restoration has been presented and developed based on a joint data fusion paradigm. It is worth highlighting that the Sentinel code and the dataset have been implemented from scratch and made available to interested research for further analysis and investigation.
  arXiv  Detail & Related papers  (2021-06-23T08:15:01Z)
• Pseudo-LiDAR Point Cloud Interpolation Based on 3D Motion Representation and Spatial Supervision [68.35777836993212]
  We propose a Pseudo-LiDAR point cloud network to generate temporally and spatially high-quality point cloud sequences. By exploiting the scene flow between point clouds, the proposed network is able to learn a more accurate representation of the 3D spatial motion relationship.
  arXiv  Detail & Related papers  (2020-06-20T03:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.