Unsupervised Pansharpening Based on Self-Attention Mechanism

• URL: http://arxiv.org/abs/2006.09303v3
• Date: Sun, 30 Aug 2020 11:48:18 GMT
• Title: Unsupervised Pansharpening Based on Self-Attention Mechanism
• Authors: Ying Qu, Razieh Kaviani Baghbaderani, Hairong Qi, Chiman Kwan
• Abstract summary: We propose an unsupervised pansharpening (UP) method in a deep-learning framework to address the challenges based on the self-attention mechanism (SAM) The proposed approach is able to reconstruct sharper MSI of different types, with more details and less spectral distortion as compared to the state-of-the-art.
• Score: 12.995590360954957
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pansharpening is to fuse a multispectral image (MSI) of low-spatial-resolution (LR) but rich spectral characteristics with a panchromatic image (PAN) of high-spatial-resolution (HR) but poor spectral characteristics. Traditional methods usually inject the extracted high-frequency details from PAN into the up-sampled MSI. Recent deep learning endeavors are mostly supervised assuming the HR MSI is available, which is unrealistic especially for satellite images. Nonetheless, these methods could not fully exploit the rich spectral characteristics in the MSI. Due to the wide existence of mixed pixels in satellite images where each pixel tends to cover more than one constituent material, pansharpening at the subpixel level becomes essential. In this paper, we propose an unsupervised pansharpening (UP) method in a deep-learning framework to address the above challenges based on the self-attention mechanism (SAM), referred to as UP-SAM. The contribution of this paper is three-fold. First, the self-attention mechanism is proposed where the spatial varying detail extraction and injection functions are estimated according to the attention representations indicating spectral characteristics of the MSI with sub-pixel accuracy. Second, such attention representations are derived from mixed pixels with the proposed stacked attention network powered with a stick-breaking structure to meet the physical constraints of mixed pixel formulations. Third, the detail extraction and injection functions are spatial varying based on the attention representations, which largely improves the reconstruction accuracy. Extensive experimental results demonstrate that the proposed approach is able to reconstruct sharper MSI of different types, with more details and less spectral distortion as compared to the state-of-the-art.

Related papers

• Universal Pansharpening Foundation Model [67.10467574892282]
  Pansharpening generates the high-resolution multi-spectral (MS) image by integrating spatial details from a texture-rich panchromatic (PAN) image and spectral attributes from a low-resolution MS image.<n>We present FoundPS, a universal pansharpening foundation model for satellite-agnostic and scene-robust fusion.
  arXiv  Detail & Related papers  (2026-03-04T08:30:15Z)
• SpectraMorph: Structured Latent Learning for Self-Supervised Hyperspectral Super-Resolution [0.0]
  Hyperspectral sensors capture dense spectra per pixel but suffer from low spatial resolution.<n>Co-registered companion sensors such as multispectral, RGB, or panchromatic cameras provide high-resolution spatial detail.<n>We propose SpectraMorph, a physics-guided self-supervised fusion framework with a structured latent space.
  arXiv  Detail & Related papers  (2025-10-23T17:59:26Z)
• Spatial-Spectral Binarized Neural Network for Panchromatic and Multi-spectral Images Fusion [25.15016853820625]
  Deep learning models have achieved excellent performance, but they often come with high computational complexity.<n>In this paper, we explore the feasibility of applying the binary neural network (BNN) to pan-sharpening.<n>A series of S2B-Conv form a brand-new binary network for pan-sharpening, dubbed as S2BNet.
  arXiv  Detail & Related papers  (2025-09-27T14:10:51Z)
• SpectraLift: Physics-Guided Spectral-Inversion Network for Self-Supervised Hyperspectral Image Super-Resolution [1.4425878137951234]
  Fusing high-spatial-resolution multispectral images (HR-MSI) with low-spatial-resolution hyperspectral images (LR-HSI) is a promising route to recover fine spatial structures.<n>We present SpectraLift, a fully self-supervised framework that fuses LR-HSI and HR-MSI inputs using only the MSI's Spectral Response (SRF)<n>At inference, SpectraLift uses the trained network to map the HR-MSI pixel-wise into a HR-HSI estimate.
  arXiv  Detail & Related papers  (2025-07-17T17:57:18Z)
• Breaking Spatial Boundaries: Spectral-Domain Registration Guided Hyperspectral and Multispectral Blind Fusion [14.285239151249193]
  The blind fusion of unregistered hyperspectral images (HSIs) and multispectral images (MSIs) has attracted growing attention recently.<n>To address the registration challenge, most existing methods employ spatial transformations on the HSI to achieve alignment with the MSI.<n>We propose tackling the registration problem from the spectral domain.
  arXiv  Detail & Related papers  (2025-06-25T10:00:51Z)
• From Image- to Pixel-level: Label-efficient Hyperspectral Image Reconstruction [9.181668145020895]
  We introduce a pixel-level spectral super-resolution (Pixel-SSR) paradigm that reconstructs hyperspectral images from RGB and point spectra. Despite its advantages, Pixel-SSR presents two key challenges: 1) generalizability to novel scenes lacking point spectra, and 2) effective information extraction to promote reconstruction accuracy.
  arXiv  Detail & Related papers  (2025-03-10T02:23:32Z)
• Multi-Head Attention Residual Unfolded Network for Model-Based Pansharpening [2.874893537471256]
  Unfolding fusion methods integrate the powerful representation capabilities of deep learning with the robustness of model-based approaches. In this paper, we propose a model-based deep unfolded method for satellite image fusion. Experimental results on PRISMA, Quickbird, and WorldView2 datasets demonstrate the superior performance of our method.
  arXiv  Detail & Related papers  (2024-09-04T13:05:00Z)
• HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Model [88.13261547704444]
  Hyper SIGMA is a vision transformer-based foundation model for HSI interpretation. It integrates spatial and spectral features using a specially designed spectral enhancement module. It shows significant advantages in scalability, robustness, cross-modal transferring capability, and real-world applicability.
  arXiv  Detail & Related papers  (2024-06-17T13:22:58Z)
• CMT: Cross Modulation Transformer with Hybrid Loss for Pansharpening [14.459280238141849]
  Pansharpening aims to enhance remote sensing image (RSI) quality by merging high-resolution panchromatic (PAN) with multispectral (MS) images. Prior techniques struggled to optimally fuse PAN and MS images for enhanced spatial and spectral information. We present the Cross Modulation Transformer (CMT), a pioneering method that modifies the attention mechanism.
  arXiv  Detail & Related papers  (2024-04-01T13:55:44Z)
• ESSAformer: Efficient Transformer for Hyperspectral Image Super-resolution [76.7408734079706]
  Single hyperspectral image super-resolution (single-HSI-SR) aims to restore a high-resolution hyperspectral image from a low-resolution observation. We propose ESSAformer, an ESSA attention-embedded Transformer network for single-HSI-SR with an iterative refining structure.
  arXiv  Detail & Related papers  (2023-07-26T07:45:14Z)
• Unsupervised Spectral Demosaicing with Lightweight Spectral Attention Networks [6.7433262627741914]
  This paper presents a deep learning-based spectral demosaicing technique trained in an unsupervised manner. The proposed method outperforms conventional unsupervised methods in terms of spatial distortion suppression, spectral fidelity, robustness, and computational cost.
  arXiv  Detail & Related papers  (2023-07-05T02:45:44Z)
• PC-GANs: Progressive Compensation Generative Adversarial Networks for Pan-sharpening [50.943080184828524]
  We propose a novel two-step model for pan-sharpening that sharpens the MS image through the progressive compensation of the spatial and spectral information. The whole model is composed of triple GANs, and based on the specific architecture, a joint compensation loss function is designed to enable the triple GANs to be trained simultaneously.
  arXiv  Detail & Related papers  (2022-07-29T03:09:21Z)
• Decoupled-and-Coupled Networks: Self-Supervised Hyperspectral Image Super-Resolution with Subpixel Fusion [67.35540259040806]
  We propose a subpixel-level HS super-resolution framework by devising a novel decoupled-and-coupled network, called DCNet. As the name suggests, DC-Net first decouples the input into common (or cross-sensor) and sensor-specific components. We append a self-supervised learning module behind the CSU net by guaranteeing the material consistency to enhance the detailed appearances of the restored HS product.
  arXiv  Detail & Related papers  (2022-05-07T23:40:36Z)
• Hyperspectral Image Segmentation based on Graph Processing over Multilayer Networks [51.15952040322895]
  One important task of hyperspectral image (HSI) processing is the extraction of spectral-spatial features. We propose several approaches to HSI segmentation based on M-GSP feature extraction. Our experimental results demonstrate the strength of M-GSP in HSI processing and spectral-spatial information extraction.
  arXiv  Detail & Related papers  (2021-11-29T23:28:18Z)
• Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction [127.20208645280438]
  Hyperspectral image (HSI) reconstruction aims to recover the 3D spatial-spectral signal from a 2D measurement. Modeling the inter-spectra interactions is beneficial for HSI reconstruction. Mask-guided Spectral-wise Transformer (MST) proposes a novel framework for HSI reconstruction.
  arXiv  Detail & Related papers  (2021-11-15T16:59:48Z)
• Hyperspectral Pansharpening Based on Improved Deep Image Prior and Residual Reconstruction [64.10636296274168]
  Hyperspectral pansharpening aims to synthesize a low-resolution hyperspectral image (LR-HSI) with a registered panchromatic image (PAN) to generate an enhanced HSI with high spectral and spatial resolution. Recently proposed HS pansharpening methods have obtained remarkable results using deep convolutional networks (ConvNets) We propose a novel over-complete network, called HyperKite, which focuses on learning high-level features by constraining the receptive from increasing in the deep layers.
  arXiv  Detail & Related papers  (2021-07-06T14:11:03Z)
• Spatial-Spectral Manifold Embedding of Hyperspectral Data [43.479889860715275]
  We propose a novel hyperspectral embedding approach by simultaneously considering spatial and spectral information. spatial-spectral manifold embedding (SSME) models the spatial and spectral information jointly in a patch-based fashion. SSME not only learns the spectral embedding by using the adjacency matrix obtained by similarity measurement between spectral signatures, but also models the spatial neighbours of a target pixel in hyperspectral scene.
  arXiv  Detail & Related papers  (2020-07-17T05:40:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.