Deep Multimodal Fusion for Semantic Segmentation of Remote Sensing Earth Observation Data

• URL: http://arxiv.org/abs/2410.00469v1
• Date: Tue, 1 Oct 2024 07:50:37 GMT
• Title: Deep Multimodal Fusion for Semantic Segmentation of Remote Sensing Earth Observation Data
• Authors: Ivica Dimitrovski, Vlatko Spasev, Ivan Kitanovski,
• Abstract summary: This paper proposes a late fusion deep learning model (LF-DLM) for semantic segmentation. One branch integrates detailed textures from aerial imagery captured by UNetFormer with a Multi-Axis Vision Transformer (ViT) backbone. The other branch captures complex-temporal dynamics from the Sentinel-2 satellite imageMax time series using a U-ViNet with Temporal Attention (U-TAE)
• Score: 0.08192907805418582
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Accurate semantic segmentation of remote sensing imagery is critical for various Earth observation applications, such as land cover mapping, urban planning, and environmental monitoring. However, individual data sources often present limitations for this task. Very High Resolution (VHR) aerial imagery provides rich spatial details but cannot capture temporal information about land cover changes. Conversely, Satellite Image Time Series (SITS) capture temporal dynamics, such as seasonal variations in vegetation, but with limited spatial resolution, making it difficult to distinguish fine-scale objects. This paper proposes a late fusion deep learning model (LF-DLM) for semantic segmentation that leverages the complementary strengths of both VHR aerial imagery and SITS. The proposed model consists of two independent deep learning branches. One branch integrates detailed textures from aerial imagery captured by UNetFormer with a Multi-Axis Vision Transformer (MaxViT) backbone. The other branch captures complex spatio-temporal dynamics from the Sentinel-2 satellite image time series using a U-Net with Temporal Attention Encoder (U-TAE). This approach leads to state-of-the-art results on the FLAIR dataset, a large-scale benchmark for land cover segmentation using multi-source optical imagery. The findings highlight the importance of multi-modality fusion in improving the accuracy and robustness of semantic segmentation in remote sensing applications.

Related papers

• STCMOT: Spatio-Temporal Cohesion Learning for UAV-Based Multiple Object Tracking [13.269416985959404]
  Multiple object tracking (MOT) in Unmanned Aerial Vehicle (UAV) videos is important for diverse applications in computer vision. We propose a novel Spatio-Temporal Cohesion Multiple Object Tracking framework (STCMOT) We use historical embedding features to model the representation of ReID and detection features in a sequential order. Our framework sets a new state-of-the-art performance in MOTA and IDF1 metrics.
  arXiv  Detail & Related papers  (2024-09-17T14:34:18Z)
• SatSwinMAE: Efficient Autoencoding for Multiscale Time-series Satellite Imagery [1.6180992915701702]
  We extend the SwinE model to integrate temporal information for satellite time-series data. The architecture employs a hierarchical 3D Masked Autoencoder (MAE) with Video Swin Transformer blocks. Our approach shows significant performance improvements over existing state-of-the-art foundation models for all the evaluated downstream tasks.
  arXiv  Detail & Related papers  (2024-05-03T22:55:56Z)
• Multi-view Aggregation Network for Dichotomous Image Segmentation [76.75904424539543]
  Dichotomous Image (DIS) has recently emerged towards high-precision object segmentation from high-resolution natural images. Existing methods rely on tedious multiple encoder-decoder streams and stages to gradually complete the global localization and local refinement. Inspired by it, we model DIS as a multi-view object perception problem and provide a parsimonious multi-view aggregation network (MVANet) Experiments on the popular DIS-5K dataset show that our MVANet significantly outperforms state-of-the-art methods in both accuracy and speed.
  arXiv  Detail & Related papers  (2024-04-11T03:00:00Z)
• SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation [69.42764583465508]
  We explore the potential of generative image diffusion to address the scarcity of annotated data in earth observation tasks. To the best of our knowledge, we are the first to generate both images and corresponding masks for satellite segmentation.
  arXiv  Detail & Related papers  (2024-03-25T10:30:22Z)
• FLAIR #2: textural and temporal information for semantic segmentation from multi-source optical imagery [0.0]
  This dataset includes two very distinct types of data, which are exploited for a semantic segmentation task aimed at mapping land cover. The data fusion workflow proposes the exploitation of the fine spatial and textural information of very high spatial resolution (VHR) mono-temporal aerial imagery and the temporal and spectral richness of high spatial resolution (HR) time series of Copernicus Sentinel-2 satellite images.
  arXiv  Detail & Related papers  (2023-05-23T18:47:19Z)
• Local-Global Temporal Difference Learning for Satellite Video Super-Resolution [55.69322525367221]
  We propose to exploit the well-defined temporal difference for efficient and effective temporal compensation. To fully utilize the local and global temporal information within frames, we systematically modeled the short-term and long-term temporal discrepancies. Rigorous objective and subjective evaluations conducted across five mainstream video satellites demonstrate that our method performs favorably against state-of-the-art approaches.
  arXiv  Detail & Related papers  (2023-04-10T07:04:40Z)
• SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery [74.82821342249039]
  We present SatMAE, a pre-training framework for temporal or multi-spectral satellite imagery based on Masked Autoencoder (MAE) To leverage temporal information, we include a temporal embedding along with independently masking image patches across time.
  arXiv  Detail & Related papers  (2022-07-17T01:35:29Z)
• Transformer Meets Convolution: A Bilateral Awareness Net-work for Semantic Segmentation of Very Fine Resolution Ur-ban Scene Images [6.460167724233707]
  We propose a bilateral awareness network (BANet) which contains a dependency path and a texture path. BANet captures the long-range relationships and fine-grained details in VFR images. Experiments conducted on the three large-scale urban scene image segmentation datasets, i.e., ISPRS Vaihingen dataset, ISPRS Potsdam dataset, and UAVid dataset, demonstrate the effective-ness of BANet.
  arXiv  Detail & Related papers  (2021-06-23T13:57:36Z)
• DS-Net: Dynamic Spatiotemporal Network for Video Salient Object Detection [78.04869214450963]
  We propose a novel dynamic temporal-temporal network (DSNet) for more effective fusion of temporal and spatial information. We show that the proposed method achieves superior performance than state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2020-12-09T06:42:30Z)
• Dense Attention Fluid Network for Salient Object Detection in Optical Remote Sensing Images [193.77450545067967]
  We propose an end-to-end Dense Attention Fluid Network (DAFNet) for salient object detection in optical remote sensing images (RSIs) A Global Context-aware Attention (GCA) module is proposed to adaptively capture long-range semantic context relationships. We construct a new and challenging optical RSI dataset for SOD that contains 2,000 images with pixel-wise saliency annotations.
  arXiv  Detail & Related papers  (2020-11-26T06:14:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.