Unaligned RGB Guided Hyperspectral Image Super-Resolution with Spatial-Spectral Concordance

• URL: http://arxiv.org/abs/2505.02109v1
• Date: Sun, 04 May 2025 13:29:31 GMT
• Title: Unaligned RGB Guided Hyperspectral Image Super-Resolution with Spatial-Spectral Concordance
• Authors: Yingkai Zhang, Zeqiang Lai, Tao Zhang, Ying Fu, Chenghu Zhou,
• Abstract summary: Hyperspectral images super-resolution aims to improve the spatial resolution, yet its performance is often limited at high-resolution ratios.<n>We introduce a framework for unaligned reference RGB guided HSI SR to address the issues of inaccurate alignment and poor interactivity.<n>Our method outperforms state-of-the-art approaches on both quantitative and qualitative evaluations.
• Score: 18.045877106333766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hyperspectral images super-resolution aims to improve the spatial resolution, yet its performance is often limited at high-resolution ratios. The recent adoption of high-resolution reference images for super-resolution is driven by the poor spatial detail found in low-resolution HSIs, presenting it as a favorable method. However, these approaches cannot effectively utilize information from the reference image, due to the inaccuracy of alignment and its inadequate interaction between alignment and fusion modules. In this paper, we introduce a Spatial-Spectral Concordance Hyperspectral Super-Resolution (SSC-HSR) framework for unaligned reference RGB guided HSI SR to address the issues of inaccurate alignment and poor interactivity of the previous approaches. Specifically, to ensure spatial concordance, i.e., align images more accurately across resolutions and refine textures, we construct a Two-Stage Image Alignment with a synthetic generation pipeline in the image alignment module, where the fine-tuned optical flow model can produce a more accurate optical flow in the first stage and warp model can refine damaged textures in the second stage. To enhance the interaction between alignment and fusion modules and ensure spectral concordance during reconstruction, we propose a Feature Aggregation module and an Attention Fusion module. In the feature aggregation module, we introduce an Iterative Deformable Feature Aggregation block to achieve significant feature matching and texture aggregation with the fusion multi-scale results guidance, iteratively generating learnable offset. Besides, we introduce two basic spectral-wise attention blocks in the attention fusion module to model the inter-spectra interactions. Extensive experiments on three natural or remote-sensing datasets show that our method outperforms state-of-the-art approaches on both quantitative and qualitative evaluations.

Related papers

• PIF-Net: Ill-Posed Prior Guided Multispectral and Hyperspectral Image Fusion via Invertible Mamba and Fusion-Aware LoRA [0.16385815610837165]
  The goal of multispectral and hyperspectral image fusion (MHIF) is to generate high-quality images that simultaneously possess rich spectral information and fine spatial details.<n>Previous studies have not effectively addressed the ill-posed nature caused by data misalignment.<n>We propose a fusion framework named PIF-Net, which explicitly incorporates ill-posed priors to effectively fuse multispectral images and hyperspectral images.
  arXiv  Detail & Related papers  (2025-08-01T09:17:17Z)
• A Fusion-Guided Inception Network for Hyperspectral Image Super-Resolution [4.487807378174191]
  We propose a single-image super-resolution model called the Fusion-Guided Inception Network (FGIN)<n>Specifically, we first employ a spectral-spatial fusion module to effectively integrate spectral and spatial information.<n>An Inception-like hierarchical feature extraction strategy is used to capture multiscale spatial dependencies.<n>To further enhance reconstruction quality, we incorporate an optimized upsampling module that combines bilinear with depthwise separable convolutions.
  arXiv  Detail & Related papers  (2025-05-06T11:15:59Z)
• HSRMamba: Contextual Spatial-Spectral State Space Model for Single Hyperspectral Super-Resolution [41.93421212397078]
  Mamba has demonstrated exceptional performance in visual tasks due to its powerful global modeling capabilities and linear computational complexity.<n>In HSISR, Mamba faces challenges as transforming images into 1D sequences neglects the spatial-spectral structural relationships between locally adjacent pixels.<n>We propose HSRMamba, a contextual spatial-spectral modeling state space model for HSISR, to address these issues both locally and globally.
  arXiv  Detail & Related papers  (2025-01-30T17:10:53Z)
• Unleashing Correlation and Continuity for Hyperspectral Reconstruction from RGB Images [64.80875911446937]
  We propose a Correlation and Continuity Network (CCNet) for HSI reconstruction from RGB images.<n>For the correlation of local spectrum, we introduce the Group-wise Spectral Correlation Modeling (GrSCM) module.<n>For the continuity of global spectrum, we design the Neighborhood-wise Spectral Continuity Modeling (NeSCM) module.
  arXiv  Detail & Related papers  (2025-01-02T15:14:40Z)
• Physics-Inspired Degradation Models for Hyperspectral Image Fusion [61.743696362028246]
  Most fusion methods solely focus on the fusion algorithm itself and overlook the degradation models. We propose physics-inspired degradation models (PIDM) to model the degradation of LR-HSI and HR-MSI. Our proposed PIDM can boost the fusion performance of existing fusion methods in practical scenarios.
  arXiv  Detail & Related papers  (2024-02-04T09:07:28Z)
• Unsupervised Hyperspectral and Multispectral Images Fusion Based on the Cycle Consistency [21.233354336608205]
  We propose an unsupervised HSI and MSI fusion model based on the cycle consistency, called CycFusion. The CycFusion learns the domain transformation between low spatial resolution HSI (LrHSI) and high spatial resolution MSI (HrMSI) Experiments conducted on several datasets show that our proposed model outperforms all compared unsupervised fusion methods.
  arXiv  Detail & Related papers  (2023-07-07T06:47:15Z)
• Hyperspectral Image Super Resolution with Real Unaligned RGB Guidance [11.711656319221072]
  We propose an HSI fusion network with heterogenous feature extractions, multi-stage feature alignments, and attentive feature fusion. Our method obtains a clear improvement over existing single-image and fusion-based super-resolution methods on quantitative assessment as well as visual comparison.
  arXiv  Detail & Related papers  (2023-02-13T11:56:45Z)
• Decoupled-and-Coupled Networks: Self-Supervised Hyperspectral Image Super-Resolution with Subpixel Fusion [67.35540259040806]
  We propose a subpixel-level HS super-resolution framework by devising a novel decoupled-and-coupled network, called DCNet. As the name suggests, DC-Net first decouples the input into common (or cross-sensor) and sensor-specific components. We append a self-supervised learning module behind the CSU net by guaranteeing the material consistency to enhance the detailed appearances of the restored HS product.
  arXiv  Detail & Related papers  (2022-05-07T23:40:36Z)
• Enhanced Hyperspectral Image Super-Resolution via RGB Fusion and TV-TV Minimization [9.584717030078245]
  Hyperspectral (HS) images contain detailed spectral information that has proven crucial in applications like remote sensing, surveillance, and astronomy. Because of hardware limitations of HS cameras, the captured images have low spatial resolution. To improve them, the low-resolution hyperspectral images are fused with conventional high-resolution RGB images via a technique known as fusion based HS image super-resolution.
  arXiv  Detail & Related papers  (2021-06-13T18:52:47Z)
• Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses [67.01164492518481]
  This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses. We propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input. Our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission.
  arXiv  Detail & Related papers  (2021-02-14T06:44:47Z)
• RGB-D Salient Object Detection with Cross-Modality Modulation and Selection [126.4462739820643]
  We present an effective method to progressively integrate and refine the cross-modality complementarities for RGB-D salient object detection (SOD) The proposed network mainly solves two challenging issues: 1) how to effectively integrate the complementary information from RGB image and its corresponding depth map, and 2) how to adaptively select more saliency-related features.
  arXiv  Detail & Related papers  (2020-07-14T14:22:50Z)
• Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization [99.96632216070718]
  Light field (LF) images acquired by hand-held devices usually suffer from low spatial resolution. The high-dimensional spatiality characteristic and complex geometrical structure of LF images make the problem more challenging than traditional single-image SR. We propose a novel learning-based LF framework, in which each view of an LF image is first individually super-resolved.
  arXiv  Detail & Related papers  (2020-04-05T14:39:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.