Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral
Compressive Imaging
- URL: http://arxiv.org/abs/2205.10102v1
- Date: Fri, 20 May 2022 11:37:44 GMT
- Title: Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral
Compressive Imaging
- Authors: Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Henghui Ding, Yulun
Zhang, Radu Timofte, Luc Van Gool
- Abstract summary: We propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration.
By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST) for HSI reconstruction.
- Score: 142.11622043078867
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In coded aperture snapshot spectral compressive imaging (CASSI) systems,
hyperspectral image (HSI) reconstruction methods are employed to recover the
spatial-spectral signal from a compressed measurement. Among these algorithms,
deep unfolding methods demonstrate promising performance but suffer from two
issues. Firstly, they do not estimate the degradation patterns and
ill-posedness degree from the highly related CASSI to guide the iterative
learning. Secondly, they are mainly CNN-based, showing limitations in capturing
long-range dependencies. In this paper, we propose a principled
Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the
compressed image and physical mask, and then uses these parameters to control
each iteration. Moreover, we customize a novel Half-Shuffle Transformer (HST)
that simultaneously captures local contents and non-local dependencies. By
plugging HST into DAUF, we establish the first Transformer-based deep unfolding
method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI
reconstruction. Experiments show that DAUHST significantly surpasses
state-of-the-art methods while requiring cheaper computational and memory
costs. Code and models will be released to the public.
Related papers
- Coarse-Fine Spectral-Aware Deformable Convolution For Hyperspectral Image Reconstruction [15.537910100051866]
We study the inverse problem of Coded Aperture Snapshot Spectral Imaging (CASSI)
We propose Coarse-Fine Spectral-Aware Deformable Convolution Network (CFSDCN)
Our CFSDCN significantly outperforms previous state-of-the-art (SOTA) methods on both simulated and real HSI datasets.
arXiv Detail & Related papers (2024-06-18T15:15:12Z) - Look-Around Before You Leap: High-Frequency Injected Transformer for Image Restoration [46.96362010335177]
In this paper, we propose HIT, a simple yet effective High-frequency Injected Transformer for image restoration.
Specifically, we design a window-wise injection module (WIM), which incorporates abundant high-frequency details into the feature map, to provide reliable references for restoring high-quality images.
In addition, we introduce a spatial enhancement unit (SEU) to preserve essential spatial relationships that may be lost due to the computations carried out across channel dimensions in the BIM.
arXiv Detail & Related papers (2024-03-30T08:05:00Z) - Pixel Adaptive Deep Unfolding Transformer for Hyperspectral Image
Reconstruction [58.32266851510948]
We propose a Pixel Adaptive Deep Unfolding Transformer (PADUT) for HSI reconstruction.
In the data module, a pixel adaptive descent step is employed to focus on pixel-level degradation.
In the prior module, we introduce the Non-local Spectral Transformer (NST) to emphasize the 3D characteristics of HSI for recovering.
arXiv Detail & Related papers (2023-08-21T16:12:31Z) - Unfolding Framework with Prior of Convolution-Transformer Mixture and
Uncertainty Estimation for Video Snapshot Compressive Imaging [7.601695814245209]
We consider the problem of video snapshot compressive imaging (SCI), where sequential high-speed frames are modulated by different masks and captured by a single measurement.
By combining optimization algorithms and neural networks, deep unfolding networks (DUNs) score tremendous achievements in solving inverse problems.
arXiv Detail & Related papers (2023-06-20T06:25:48Z) - Degradation-Noise-Aware Deep Unfolding Transformer for Hyperspectral
Image Denoising [9.119226249676501]
Hyperspectral images (HSIs) are often quite noisy because of narrow band spectral filtering.
To reduce the noise in HSI data cubes, both model-driven and learning-based denoising algorithms have been proposed.
This paper proposes a Degradation-Noise-Aware Unfolding Network (DNA-Net) that addresses these issues.
arXiv Detail & Related papers (2023-05-06T13:28:20Z) - MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral
Reconstruction [148.26195175240923]
We propose a novel Transformer-based method, Multi-stage Spectral-wise Transformer (MST++) for efficient spectral reconstruction.
In the NTIRE 2022 Spectral Reconstruction Challenge, our approach won the First place.
arXiv Detail & Related papers (2022-04-17T02:39:32Z) - Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction [138.04956118993934]
We propose a novel Transformer-based method, coarse-to-fine sparse Transformer (CST)
CST embedding HSI sparsity into deep learning for HSI reconstruction.
In particular, CST uses our proposed spectra-aware screening mechanism (SASM) for coarse patch selecting. Then the selected patches are fed into our customized spectra-aggregation hashing multi-head self-attention (SAH-MSA) for fine pixel clustering and self-similarity capturing.
arXiv Detail & Related papers (2022-03-09T16:17:47Z) - CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning.
The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery.
The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
arXiv Detail & Related papers (2021-12-31T04:37:11Z) - Spatially-Adaptive Image Restoration using Distortion-Guided Networks [51.89245800461537]
We present a learning-based solution for restoring images suffering from spatially-varying degradations.
We propose SPAIR, a network design that harnesses distortion-localization information and dynamically adjusts to difficult regions in the image.
arXiv Detail & Related papers (2021-08-19T11:02:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.