Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations

• URL: http://arxiv.org/abs/2405.17818v1
• Date: Tue, 28 May 2024 04:29:23 GMT
• Title: Hyperspectral and multispectral image fusion with arbitrary resolution through self-supervised representations
• Authors: Ting Wang, Zipei Yan, Jizhou Li, Xile Zhao, Chao Wang, Michael Ng,
• Abstract summary: The fusion of a low-resolution hyperspectral image with a high-resolution multispectral image (HR-MSI) has emerged as an effective technique for achieving HSI super-resolution (SR) Previous studies have mainly concentrated on estimating the posterior distribution of the latent high-resolution hyperspectral image (HR-HSI) We propose a novel continuous low-rank factorization (CLoRF) by integrating two neural representations into the matrix factorization.
• Score: 23.04458119996
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The fusion of a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) has emerged as an effective technique for achieving HSI super-resolution (SR). Previous studies have mainly concentrated on estimating the posterior distribution of the latent high-resolution hyperspectral image (HR-HSI), leveraging an appropriate image prior and likelihood computed from the discrepancy between the latent HSI and observed images. Low rankness stands out for preserving latent HSI characteristics through matrix factorization among the various priors. However, this method only enhances resolution within the dimensions of the two modalities. To overcome this limitation, we propose a novel continuous low-rank factorization (CLoRF) by integrating two neural representations into the matrix factorization, capturing spatial and spectral information, respectively. This approach enables us to harness both the low rankness from the matrix factorization and the continuity from neural representation in a self-supervised manner. Theoretically, we prove the low-rank property and Lipschitz continuity in the proposed continuous low-rank factorization. Experimentally, our method significantly surpasses existing techniques and achieves user-desired resolutions without the need for neural network retraining.

Related papers

• Reconstructing Interpretable Features in Computational Super-Resolution microscopy via Regularized Latent Search [2.7194314957925094]
  Supervised deep learning approaches can artificially increase the resolution of microscopy images by learning a mapping between two image resolutions or modalities. Recent methods based on GAN latent search offered a drastic increase in resolution without the need of paired images. Here, we propose a robust super-resolution method based on regularized latent search(RLS) that offers an actionable balance between fidelity to the ground-truth and realism of the recovered image.
  arXiv  Detail & Related papers  (2024-05-29T14:20:46Z)
• Hyperspectral Image Fusion via Logarithmic Low-rank Tensor Ring Decomposition [26.76968345244154]
  We study the low-rankness of TR factors from the TNN perspective and consider the mode-2 logarithmic TNN (LTNN) on each TR factor. A novel fusion model is proposed by incorporating this LTNN regularization and the weighted total variation.
  arXiv  Detail & Related papers  (2023-10-16T04:02:34Z)
• ESSAformer: Efficient Transformer for Hyperspectral Image Super-resolution [76.7408734079706]
  Single hyperspectral image super-resolution (single-HSI-SR) aims to restore a high-resolution hyperspectral image from a low-resolution observation. We propose ESSAformer, an ESSA attention-embedded Transformer network for single-HSI-SR with an iterative refining structure.
  arXiv  Detail & Related papers  (2023-07-26T07:45:14Z)
• An Optimization-based Deep Equilibrium Model for Hyperspectral Image Deconvolution with Convergence Guarantees [71.57324258813675]
  We propose a novel methodology for addressing the hyperspectral image deconvolution problem. A new optimization problem is formulated, leveraging a learnable regularizer in the form of a neural network. The derived iterative solver is then expressed as a fixed-point calculation problem within the Deep Equilibrium framework.
  arXiv  Detail & Related papers  (2023-06-10T08:25:16Z)
• Generalized Expectation Maximization Framework for Blind Image Super Resolution [28.108363151431877]
  We propose an end-to-end learning framework for the blind SISR problem. The proposed method integrates learning techniques into the generalized expectation-maximization (GEM) algorithm and infers HR images from the maximum likelihood estimation (MLE)
  arXiv  Detail & Related papers  (2023-05-23T10:01:58Z)
• Semantic Encoder Guided Generative Adversarial Face Ultra-Resolution Network [15.102899995465041]
  We propose a novel face super-resolution method, namely Semantic guided Generative Adversarial Face Ultra-Resolution Network (SEGA-FURN) The proposed network is composed of a novel semantic encoder that has the ability to capture the embedded semantics to guide adversarial learning and a novel generator that uses a hierarchical architecture named Residual in Internal Block (RIDB) Experiments on large face datasets have proved that the proposed method can achieve superior super-resolution results and significantly outperform other state-of-the-art methods in both qualitative and quantitative comparisons.
  arXiv  Detail & Related papers  (2022-11-18T23:16:57Z)
• Quality Assessment of Image Super-Resolution: Balancing Deterministic and Statistical Fidelity [14.586878663223832]
  We look at the problem of SR image quality assessment (SR IQA) in a two-dimensional (2D) space of deterministic fidelity (DF) versus statistical fidelity (SF) We propose an uncertainty weighting scheme that merges the two fidelity measures into an overall quality prediction named the Super Resolution Image Fidelity (SRIF) index.
  arXiv  Detail & Related papers  (2022-07-15T02:09:17Z)
• HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging [138.04956118993934]
  We propose a high-resolution dual-domain learning network (HDNet) for HSI reconstruction. On the one hand, the proposed HR spatial-spectral attention module with its efficient feature fusion provides continuous and fine pixel-level features. On the other hand, frequency domain learning (FDL) is introduced for HSI reconstruction to narrow the frequency domain discrepancy.
  arXiv  Detail & Related papers  (2022-03-04T06:37:45Z)
• Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling [139.25215100378284]
  We propose a hierarchical conditional flow (HCFlow) as a unified framework for image SR and image rescaling. HCFlow learns a mapping between HR and LR image pairs by modelling the distribution of the LR image and the rest high-frequency component simultaneously. To further enhance the performance, other losses such as perceptual loss and GAN loss are combined with the commonly used negative log-likelihood loss in training.
  arXiv  Detail & Related papers  (2021-08-11T16:11:01Z)
• Hyperspectral Image Super-resolution via Deep Progressive Zero-centric Residual Learning [62.52242684874278]
  Cross-modality distribution of spatial and spectral information makes the problem challenging. We propose a novel textitlightweight deep neural network-based framework, namely PZRes-Net. Our framework learns a high resolution and textitzero-centric residual image, which contains high-frequency spatial details of the scene.
  arXiv  Detail & Related papers  (2020-06-18T06:32:11Z)
• Hyperspectral-Multispectral Image Fusion with Weighted LASSO [68.04032419397677]
  We propose an approach for fusing hyperspectral and multispectral images to provide high-quality hyperspectral output. We demonstrate that the proposed sparse fusion and reconstruction provides quantitatively superior results when compared to existing methods on publicly available images.
  arXiv  Detail & Related papers  (2020-03-15T23:07:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.