CSAKD: Knowledge Distillation with Cross Self-Attention for Hyperspectral and Multispectral Image Fusion

• URL: http://arxiv.org/abs/2406.19666v1
• Date: Fri, 28 Jun 2024 05:25:57 GMT
• Title: CSAKD: Knowledge Distillation with Cross Self-Attention for Hyperspectral and Multispectral Image Fusion
• Authors: Chih-Chung Hsu, Chih-Chien Ni, Chia-Ming Lee, Li-Wei Kang,
• Abstract summary: This paper introduces a novel knowledge distillation (KD) framework for HR-MSI/LR-HSI fusion to achieve SR of LR-HSI. To fully exploit the spatial and spectral feature representations of LR-HSI and HR-MSI, we propose a novel Cross Self-Attention (CSA) fusion module. Our experimental results demonstrate that the student model achieves comparable or superior LR-HSI SR performance.
• Score: 9.3350274016294
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyperspectral imaging, capturing detailed spectral information for each pixel, is pivotal in diverse scientific and industrial applications. Yet, the acquisition of high-resolution (HR) hyperspectral images (HSIs) often needs to be addressed due to the hardware limitations of existing imaging systems. A prevalent workaround involves capturing both a high-resolution multispectral image (HR-MSI) and a low-resolution (LR) HSI, subsequently fusing them to yield the desired HR-HSI. Although deep learning-based methods have shown promising in HR-MSI/LR-HSI fusion and LR-HSI super-resolution (SR), their substantial model complexities hinder deployment on resource-constrained imaging devices. This paper introduces a novel knowledge distillation (KD) framework for HR-MSI/LR-HSI fusion to achieve SR of LR-HSI. Our KD framework integrates the proposed Cross-Layer Residual Aggregation (CLRA) block to enhance efficiency for constructing Dual Two-Streamed (DTS) network structure, designed to extract joint and distinct features from LR-HSI and HR-MSI simultaneously. To fully exploit the spatial and spectral feature representations of LR-HSI and HR-MSI, we propose a novel Cross Self-Attention (CSA) fusion module to adaptively fuse those features to improve the spatial and spectral quality of the reconstructed HR-HSI. Finally, the proposed KD-based joint loss function is employed to co-train the teacher and student networks. Our experimental results demonstrate that the student model not only achieves comparable or superior LR-HSI SR performance but also significantly reduces the model-size and computational requirements. This marks a substantial advancement over existing state-of-the-art methods. The source code is available at https://github.com/ming053l/CSAKD.

Related papers

• Test-time Training for Hyperspectral Image Super-resolution [95.38382633281398]
  Hyperspectral image (HSI) super-resolution (SR) is still lagging behind the research of RGB image SR. In this work, we propose a new test-time training method to tackle this problem. Specifically, a novel self-training framework is developed, where more accurate pseudo-labels and more accurate LR-HR relationships are generated.
  arXiv  Detail & Related papers  (2024-09-13T09:30:19Z)
• HSR-KAN: Efficient Hyperspectral Image Super-Resolution via Kolmogorov-Arnold Networks [0.16385815610837165]
  We propose an efficient HSI super-resolution (HSI-SR) model to fuse a low-resolution HSI and a high-resolution multispectral image (HR-MSI) To achieve the effective integration of spatial information from HR-MSI, we design a fusion module based on KANs. As a channel attention module integrated with KANs, KAN-CAB enables networks to accurately simulate details of spectral sequences and spatial textures.
  arXiv  Detail & Related papers  (2024-08-24T02:51:51Z)
• UnmixingSR: Material-aware Network with Unsupervised Unmixing as Auxiliary Task for Hyperspectral Image Super-resolution [5.167168688234238]
  This paper proposes a component-aware hyperspectral image (HIS) super-resolution network called UnmixingSR. We use the bond between LR abundances and HR abundances to boost the stability of our method in solving SR problems. Experimental results show that unmixing process as an auxiliary task incorporated into the SR problem is feasible and rational.
  arXiv  Detail & Related papers  (2024-07-09T03:41:02Z)
• Learning Many-to-Many Mapping for Unpaired Real-World Image Super-resolution and Downscaling [60.80788144261183]
  We propose an image downscaling and SR model dubbed as SDFlow, which simultaneously learns a bidirectional many-to-many mapping between real-world LR and HR images unsupervisedly. Experimental results on real-world image SR datasets indicate that SDFlow can generate diverse realistic LR and SR images both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-10-08T01:48:34Z)
• Unsupervised Hyperspectral and Multispectral Images Fusion Based on the Cycle Consistency [21.233354336608205]
  We propose an unsupervised HSI and MSI fusion model based on the cycle consistency, called CycFusion. The CycFusion learns the domain transformation between low spatial resolution HSI (LrHSI) and high spatial resolution MSI (HrMSI) Experiments conducted on several datasets show that our proposed model outperforms all compared unsupervised fusion methods.
  arXiv  Detail & Related papers  (2023-07-07T06:47:15Z)
• HSR-Diff:Hyperspectral Image Super-Resolution via Conditional Diffusion Models [10.865272587124027]
  We propose an HSI Super-resolution (SR) approach with Conditional Diffusion Models (HSR-Diff) HSR-Diff generates an HR-HSI via repeated refinement, in which the HR-HSI is spatial with pure Gaussian noise and iteratively refined. In addition, a progressive learning strategy is employed to exploit the global information of full-resolution images.
  arXiv  Detail & Related papers  (2023-06-21T08:04:30Z)
• Deep Posterior Distribution-based Embedding for Hyperspectral Image Super-resolution [75.24345439401166]
  This paper focuses on how to embed the high-dimensional spatial-spectral information of hyperspectral (HS) images efficiently and effectively. We formulate HS embedding as an approximation of the posterior distribution of a set of carefully-defined HS embedding events. Then, we incorporate the proposed feature embedding scheme into a source-consistent super-resolution framework that is physically-interpretable. Experiments over three common benchmark datasets demonstrate that PDE-Net achieves superior performance over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-05-30T06:59:01Z)
• HDNet: High-resolution Dual-domain Learning for Spectral Compressive Imaging [138.04956118993934]
  We propose a high-resolution dual-domain learning network (HDNet) for HSI reconstruction. On the one hand, the proposed HR spatial-spectral attention module with its efficient feature fusion provides continuous and fine pixel-level features. On the other hand, frequency domain learning (FDL) is introduced for HSI reconstruction to narrow the frequency domain discrepancy.
  arXiv  Detail & Related papers  (2022-03-04T06:37:45Z)
• An Arbitrary Scale Super-Resolution Approach for 3-Dimensional Magnetic Resonance Image using Implicit Neural Representation [37.43985628701494]
  High Resolution (HR) medical images provide rich anatomical structure details to facilitate early and accurate diagnosis. Recent studies showed that, with deep convolutional neural networks, isotropic HR MR images could be recovered from low-resolution (LR) input. We propose ArSSR, an Arbitrary Scale Super-Resolution approach for recovering 3D HR MR images.
  arXiv  Detail & Related papers  (2021-10-27T14:48:54Z)
• Hyperspectral Image Super-Resolution with Spectral Mixup and Heterogeneous Datasets [99.92564298432387]
  This work studies Hyperspectral image (HSI) super-resolution (SR) HSI SR is characterized by high-dimensional data and a limited amount of training examples. This exacerbates the undesirable behaviors of neural networks such as memorization and sensitivity to out-of-distribution samples.
  arXiv  Detail & Related papers  (2021-01-19T12:19:53Z)
• Cross-Attention in Coupled Unmixing Nets for Unsupervised Hyperspectral Super-Resolution [79.97180849505294]
  We propose a novel coupled unmixing network with a cross-attention mechanism, CUCaNet, to enhance the spatial resolution of HSI. Experiments are conducted on three widely-used HS-MS datasets in comparison with state-of-the-art HSI-SR models.
  arXiv  Detail & Related papers  (2020-07-10T08:08:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.