D$^\text{2}$UF: Deep Coded Aperture Design and Unrolling Algorithm for Compressive Spectral Image Fusion

• URL: http://arxiv.org/abs/2205.12158v1
• Date: Tue, 24 May 2022 15:39:34 GMT
• Title: D$^\text{2}$UF: Deep Coded Aperture Design and Unrolling Algorithm for Compressive Spectral Image Fusion
• Authors: Roman Jacome, Jorge Bacca and Henry Arguello
• Abstract summary: This paper presents the fusion of the compressive measurements of a low-spatial high-spectral resolution coded aperture snapshot spectral imager (CASSI) architecture and a high-spatial low-spectral resolution multispectral color filter array (MCFA) system. Unlike previous CSIF works, this paper proposes joint optimization of the sensing architectures and a reconstruction network in an end-to-end (E2E) manner.
• Score: 22.0246327137227
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Compressive spectral imaging (CSI) has attracted significant attention since it employs synthetic apertures to codify spatial and spectral information, sensing only 2D projections of the 3D spectral image. However, these optical architectures suffer from a trade-off between the spatial and spectral resolution of the reconstructed image due to technology limitations. To overcome this issue, compressive spectral image fusion (CSIF) employs the projected measurements of two CSI architectures with different resolutions to estimate a high-spatial high-spectral resolution. This work presents the fusion of the compressive measurements of a low-spatial high-spectral resolution coded aperture snapshot spectral imager (CASSI) architecture and a high-spatial low-spectral resolution multispectral color filter array (MCFA) system. Unlike previous CSIF works, this paper proposes joint optimization of the sensing architectures and a reconstruction network in an end-to-end (E2E) manner. The trainable optical parameters are the coded aperture (CA) in the CASSI and the colored coded aperture in the MCFA system, employing a sigmoid activation function and regularization function to encourage binary values on the trainable variables for an implementation purpose. Additionally, an unrolling-based network inspired by the alternating direction method of multipliers (ADMM) optimization is formulated to address the reconstruction step and the acquisition systems design jointly. Finally, a spatial-spectral inspired loss function is employed at the end of each unrolling layer to increase the convergence of the unrolling network. The proposed method outperforms previous CSIF methods, and experimental results validate the method with real measurements.

Related papers

• Physics-Inspired Degradation Models for Hyperspectral Image Fusion [61.743696362028246]
  Most fusion methods solely focus on the fusion algorithm itself and overlook the degradation models. We propose physics-inspired degradation models (PIDM) to model the degradation of LR-HSI and HR-MSI. Our proposed PIDM can boost the fusion performance of existing fusion methods in practical scenarios.
  arXiv  Detail & Related papers  (2024-02-04T09:07:28Z)
• SSIF: Learning Continuous Image Representation for Spatial-Spectral Super-Resolution [73.46167948298041]
  We propose a neural implicit model that represents an image as a function of both continuous pixel coordinates in the spatial domain and continuous wavelengths in the spectral domain. We show that SSIF generalizes well to both unseen spatial resolutions and spectral resolutions. It can generate high-resolution images that improve the performance of downstream tasks by 1.7%-7%.
  arXiv  Detail & Related papers  (2023-09-30T15:23:30Z)
• Aperture Diffraction for Compact Snapshot Spectral Imaging [27.321750056840706]
  We demonstrate a compact, cost-effective snapshot spectral imaging system named Aperture Diffraction Imaging Spectrometer (ADIS) A new optical design that each point in the object space is multiplexed to discrete encoding locations on the mosaic filter sensor is introduced. The Cascade Shift-Shuffle Spectral Transformer (CSST) with strong perception of the diffraction degeneration is designed to solve a sparsity-constrained inverse problem.
  arXiv  Detail & Related papers  (2023-09-27T16:48:46Z)
• ESSAformer: Efficient Transformer for Hyperspectral Image Super-resolution [76.7408734079706]
  Single hyperspectral image super-resolution (single-HSI-SR) aims to restore a high-resolution hyperspectral image from a low-resolution observation. We propose ESSAformer, an ESSA attention-embedded Transformer network for single-HSI-SR with an iterative refining structure.
  arXiv  Detail & Related papers  (2023-07-26T07:45:14Z)
• Residual Degradation Learning Unfolding Framework with Mixing Priors across Spectral and Spatial for Compressive Spectral Imaging [29.135848304404533]
  coded aperture snapshot spectral imaging (CASSI) is proposed. core problem of the CASSI system is to recover the reliable and fine underlying 3D spectral cube from the 2D measurement. We propose a Residual Degradation Learning Unfolding Framework (RDLUF) which bridges the gap between the sensing matrix and the degradation process.
  arXiv  Detail & Related papers  (2022-11-13T12:31:49Z)
• S^2-Transformer for Mask-Aware Hyperspectral Image Reconstruction [48.83280067393851]
  A representative hyperspectral image acquisition procedure conducts a 3D-to-2D encoding by the coded aperture snapshot spectral imager (CASSI) Two major challenges stand in the way of a high-fidelity reconstruction: (i) To obtain 2D measurements, CASSI dislocates multiple channels by disperser-titling and squeezes them onto the same spatial region, yielding an entangled data loss. We propose a spatial-spectral (S2-) transformer architecture with a mask-aware learning strategy to tackle these challenges.
  arXiv  Detail & Related papers  (2022-09-24T19:26:46Z)
• PC-GANs: Progressive Compensation Generative Adversarial Networks for Pan-sharpening [50.943080184828524]
  We propose a novel two-step model for pan-sharpening that sharpens the MS image through the progressive compensation of the spatial and spectral information. The whole model is composed of triple GANs, and based on the specific architecture, a joint compensation loss function is designed to enable the triple GANs to be trained simultaneously.
  arXiv  Detail & Related papers  (2022-07-29T03:09:21Z)
• Deep Coding Patterns Design for Compressive Near-Infrared Spectral Classification [80.93625278357229]
  spectral classification can be performed directly in the compressive domain, considering the amount of spectral information embedded in the measurements. This work proposes an end-to-end approach to jointly design the coding patterns used in CSI and the network parameters to perform spectral classification directly from the embedded near-infrared compressive measurements.
  arXiv  Detail & Related papers  (2022-05-27T15:55:53Z)
• LADMM-Net: An Unrolled Deep Network For Spectral Image Fusion From Compressive Data [6.230751621285322]
  Hyperspectral (HS) and multispectral (MS) image fusion aims at estimating a high-resolution spectral image from a low-spatial-resolution HS image and a low-spectral-resolution MS image. In this work, a deep learning architecture under the algorithm unrolling approach is proposed for solving the fusion problem from HS and MS compressive measurements.
  arXiv  Detail & Related papers  (2021-03-01T12:04:42Z)
• Spectral Response Function Guided Deep Optimization-driven Network for Spectral Super-resolution [20.014293172511074]
  This paper proposes an optimization-driven convolutional neural network (CNN) with a deep spatial-spectral prior. Experiments on two types of datasets, including natural and remote sensing images, demonstrate the spectral enhancement effect of the proposed method.
  arXiv  Detail & Related papers  (2020-11-19T07:52:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.