Test-time Training for Hyperspectral Image Super-resolution

• URL: http://arxiv.org/abs/2409.08667v1
• Date: Fri, 13 Sep 2024 09:30:19 GMT
• Title: Test-time Training for Hyperspectral Image Super-resolution
• Authors: Ke Li, Luc Van Gool, Dengxin Dai,
• Abstract summary: Hyperspectral image (HSI) super-resolution (SR) is still lagging behind the research of RGB image SR. In this work, we propose a new test-time training method to tackle this problem. Specifically, a novel self-training framework is developed, where more accurate pseudo-labels and more accurate LR-HR relationships are generated.
• Score: 95.38382633281398
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The progress on Hyperspectral image (HSI) super-resolution (SR) is still lagging behind the research of RGB image SR. HSIs usually have a high number of spectral bands, so accurately modeling spectral band interaction for HSI SR is hard. Also, training data for HSI SR is hard to obtain so the dataset is usually rather small. In this work, we propose a new test-time training method to tackle this problem. Specifically, a novel self-training framework is developed, where more accurate pseudo-labels and more accurate LR-HR relationships are generated so that the model can be further trained with them to improve performance. In order to better support our test-time training method, we also propose a new network architecture to learn HSI SR without modeling spectral band interaction and propose a new data augmentation method Spectral Mixup to increase the diversity of the training data at test time. We also collect a new HSI dataset with a diverse set of images of interesting objects ranging from food to vegetation, to materials, and to general scenes. Extensive experiments on multiple datasets show that our method can improve the performance of pre-trained models significantly after test-time training and outperform competing methods significantly for HSI SR.

Related papers

• EigenSR: Eigenimage-Bridged Pre-Trained RGB Learners for Single Hyperspectral Image Super-Resolution [15.459253235077375]
  Single hyperspectral image super-resolution (single-HSI-SR) aims to improve the resolution of a single input low-resolution HSI. Due to the bottleneck of data scarcity, single-HSI-SR lags far behind that of RGB natural images. We propose a new framework that first fine-tunes the pre-trained model with the spatial components, and then infers on unseen HSI using an iterative spectral regularization.
  arXiv  Detail & Related papers  (2024-09-06T06:46:01Z)
• Rethinking Image Super-Resolution from Training Data Perspectives [54.28824316574355]
  We investigate the understudied effect of the training data used for image super-resolution (SR) With this, we propose an automated image evaluation pipeline. We find that datasets with (i) low compression artifacts, (ii) high within-image diversity as judged by the number of different objects, and (iii) a large number of images from ImageNet or PASS all positively affect SR performance.
  arXiv  Detail & Related papers  (2024-09-01T16:25:04Z)
• CSAKD: Knowledge Distillation with Cross Self-Attention for Hyperspectral and Multispectral Image Fusion [9.3350274016294]
  This paper introduces a novel knowledge distillation (KD) framework for HR-MSI/LR-HSI fusion to achieve SR of LR-HSI. To fully exploit the spatial and spectral feature representations of LR-HSI and HR-MSI, we propose a novel Cross Self-Attention (CSA) fusion module. Our experimental results demonstrate that the student model achieves comparable or superior LR-HSI SR performance.
  arXiv  Detail & Related papers  (2024-06-28T05:25:57Z)
• Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction [62.955327005837475]
  Image super-resolution (SR) aims to learn a mapping from low-resolution (LR) to high-resolution (HR) using paired HR-LR training images. We present an efficient test-time adaptation framework for SR, named SRTTA, which is able to quickly adapt SR models to test domains with different/unknown degradation types.
  arXiv  Detail & Related papers  (2023-10-29T13:58:57Z)
• ADASR: An Adversarial Auto-Augmentation Framework for Hyperspectral and Multispectral Data Fusion [54.668445421149364]
  Deep learning-based hyperspectral image (HSI) super-resolution aims to generate high spatial resolution HSI (HR-HSI) by fusing hyperspectral image (HSI) and multispectral image (MSI) with deep neural networks (DNNs) In this letter, we propose a novel adversarial automatic data augmentation framework ADASR that automatically optimize and augments HSI-MSI sample pairs to enrich data diversity for HSI-MSI fusion.
  arXiv  Detail & Related papers  (2023-10-11T07:30:37Z)
• Hyperspectral Pansharpening Based on Improved Deep Image Prior and Residual Reconstruction [64.10636296274168]
  Hyperspectral pansharpening aims to synthesize a low-resolution hyperspectral image (LR-HSI) with a registered panchromatic image (PAN) to generate an enhanced HSI with high spectral and spatial resolution. Recently proposed HS pansharpening methods have obtained remarkable results using deep convolutional networks (ConvNets) We propose a novel over-complete network, called HyperKite, which focuses on learning high-level features by constraining the receptive from increasing in the deep layers.
  arXiv  Detail & Related papers  (2021-07-06T14:11:03Z)
• Hyperspectral Image Super-Resolution in Arbitrary Input-Output Band Settings [1.2148588577240576]
  Hyperspectral images (HSIs) with narrow spectral bands can capture rich spectral information, making them suitable for many computer vision tasks. One of the fundamental limitations of HSI is its low spatial resolution, and several recent works on super-resolution(SR) have been proposed to tackle this challenge. We propose a Meta-Learning-Based Super-Resolution(MLSR) model, which can take in HSI images at an arbitrary number of input bands' peak wavelengths and generate super-resolved HSIs with an arbitrary number of output bands' peak wavelengths.
  arXiv  Detail & Related papers  (2021-03-19T03:32:28Z)
• Hyperspectral Image Super-Resolution with Spectral Mixup and Heterogeneous Datasets [99.92564298432387]
  This work studies Hyperspectral image (HSI) super-resolution (SR) HSI SR is characterized by high-dimensional data and a limited amount of training examples. This exacerbates the undesirable behaviors of neural networks such as memorization and sensitivity to out-of-distribution samples.
  arXiv  Detail & Related papers  (2021-01-19T12:19:53Z)
• Unsupervised Alternating Optimization for Blind Hyperspectral Imagery Super-resolution [40.350308926790255]
  This paper proposes an unsupervised blind HSI SR method to handle blind HSI fusion problem. We first propose an alternating optimization based deep framework to estimate the degeneration models and reconstruct the latent image. Then, a meta-learning based mechanism is further proposed to pre-train the network, which can effectively improve the speed and generalization ability.
  arXiv  Detail & Related papers  (2020-12-03T07:52:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.