Deep Learning Approach for Hyperspectral Image Demosaicking, Spectral Correction and High-resolution RGB Reconstruction

• URL: http://arxiv.org/abs/2109.01403v1
• Date: Fri, 3 Sep 2021 09:50:03 GMT
• Title: Deep Learning Approach for Hyperspectral Image Demosaicking, Spectral Correction and High-resolution RGB Reconstruction
• Authors: Peichao Li, Michael Ebner, Philip Noonan, Conor Horgan, Anisha Bahl, Sebastien Ourselin, Jonathan Shapey and Tom Vercauteren
• Abstract summary: We propose a deep learning-based image demosaicking algorithm for snapshot hyperspectral images using supervised learning methods. Due to the lack of publicly available medical images, a synthetic image generation approach is proposed to simulate snapshot images from existing medical image datasets. The resulting demosaicked images are evaluated both quantitatively and qualitatively, showing clear improvements in image quality.
• Score: 3.0478210530038443
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hyperspectral imaging is one of the most promising techniques for intraoperative tissue characterisation. Snapshot mosaic cameras, which can capture hyperspectral data in a single exposure, have the potential to make a real-time hyperspectral imaging system for surgical decision-making possible. However, optimal exploitation of the captured data requires solving an ill-posed demosaicking problem and applying additional spectral corrections to recover spatial and spectral information of the image. In this work, we propose a deep learning-based image demosaicking algorithm for snapshot hyperspectral images using supervised learning methods. Due to the lack of publicly available medical images acquired with snapshot mosaic cameras, a synthetic image generation approach is proposed to simulate snapshot images from existing medical image datasets captured by high-resolution, but slow, hyperspectral imaging devices. Image reconstruction is achieved using convolutional neural networks for hyperspectral image super-resolution, followed by cross-talk and leakage correction using a sensor-specific calibration matrix. The resulting demosaicked images are evaluated both quantitatively and qualitatively, showing clear improvements in image quality compared to a baseline demosaicking method using linear interpolation. Moreover, the fast processing time of~45\,ms of our algorithm to obtain super-resolved RGB or oxygenation saturation maps per image frame for a state-of-the-art snapshot mosaic camera demonstrates the potential for its seamless integration into real-time surgical hyperspectral imaging applications.

Related papers

• A Unified Model for Compressed Sensing MRI Across Undersampling Patterns [69.19631302047569]
  Deep neural networks have shown great potential for reconstructing high-fidelity images from undersampled measurements. Our model is based on neural operators, a discretization-agnostic architecture. Our inference speed is also 1,400x faster than diffusion methods.
  arXiv  Detail & Related papers  (2024-10-05T20:03:57Z)
• A self-supervised and adversarial approach to hyperspectral demosaicking and RGB reconstruction in surgical imaging [3.426432165500852]
  Hyperspectral imaging holds promises in surgical imaging by offering biological tissue differentiation capabilities with detailed information that is invisible to the naked eye. For intra-operative guidance, real-time spectral data capture and display is mandated. Snapshot mosaic hyperspectral cameras are currently seen as the most suitable technology given this requirement. We present a self-supervised demosaicking and RGB reconstruction method that does not depend on paired high-resolution data as ground truth.
  arXiv  Detail & Related papers  (2024-07-27T15:29:35Z)
• On Sensitivity and Robustness of Normalization Schemes to Input Distribution Shifts in Automatic MR Image Diagnosis [58.634791552376235]
  Deep Learning (DL) models have achieved state-of-the-art performance in diagnosing multiple diseases using reconstructed images as input. DL models are sensitive to varying artifacts as it leads to changes in the input data distribution between the training and testing phases. We propose to use other normalization techniques, such as Group Normalization and Layer Normalization, to inject robustness into model performance against varying image artifacts.
  arXiv  Detail & Related papers  (2023-06-23T03:09:03Z)
• mHealth hyperspectral learning for instantaneous spatiospectral imaging of hemodynamics [0.2638512174804417]
  Hyperspectral learning exploits idea that a photograph is more than merely a picture and contains detailed spectral information. A small sampling of hyperspectral data enables spectrally informed learning to recover a hypercube from an RGB image. Hyperspectral learning is capable of recovering full spectroscopic resolution in the hypercube, comparable to high spectral resolutions of scientific spectrometers.
  arXiv  Detail & Related papers  (2023-03-27T15:12:10Z)
• Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: Application to surgical imaging [4.795951381086172]
  Hyperspectral imaging has the potential to improve tissue characterisation in real-time and with high-resolution. A demosaicking algorithm is required to fully recover the spatial and spectral information of the snapshot images. We present a fully unsupervised hyperspectral image demosaicking algorithm which only requires snapshot images for training purposes.
  arXiv  Detail & Related papers  (2023-02-21T18:07:14Z)
• Geometric Constraints Enable Self-Supervised Sinogram Inpainting in Sparse-View Tomography [7.416898042520079]
  Sparse-angle tomographic scans reduce radiation and accelerate data acquisition, but suffer from image artifacts and noise. Existing image processing algorithms can restore CT reconstruction quality but often require large training data sets or can not be used for truncated objects. This work presents a self-supervised projection inpainting method that allows optimizing missing projective views via gradient-based optimization.
  arXiv  Detail & Related papers  (2023-02-13T15:15:18Z)
• Spectral Bandwidth Recovery of Optical Coherence Tomography Images using Deep Learning [0.6990493129893112]
  Technological developments to increase the speed of acquisition often results in systems with a narrower spectral bandwidth, and hence a lower axial resolution. Traditionally, image-processing-based techniques have been utilized to reconstruct subsampled OCT data. In this study, we simulate reduced axial scan (A-scan) resolution by Gaussian windowing in the spectral domain and investigate the use of a learning-based approach for image feature reconstruction.
  arXiv  Detail & Related papers  (2023-01-02T02:18:32Z)
• Multi-modal Aggregation Network for Fast MR Imaging [85.25000133194762]
  We propose a novel Multi-modal Aggregation Network, named MANet, which is capable of discovering complementary representations from a fully sampled auxiliary modality. In our MANet, the representations from the fully sampled auxiliary and undersampled target modalities are learned independently through a specific network. Our MANet follows a hybrid domain learning framework, which allows it to simultaneously recover the frequency signal in the $k$-space domain.
  arXiv  Detail & Related papers  (2021-10-15T13:16:59Z)
• Adaptive Gradient Balancing for UndersampledMRI Reconstruction and Image-to-Image Translation [60.663499381212425]
  We enhance the image quality by using a Wasserstein Generative Adversarial Network combined with a novel Adaptive Gradient Balancing technique. In MRI, our method minimizes artifacts, while maintaining a high-quality reconstruction that produces sharper images than other techniques.
  arXiv  Detail & Related papers  (2021-04-05T13:05:22Z)
• Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery [79.69449412334188]
  In this paper, we investigate how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches. We introduce a spatial-spectral prior network (SSPN) to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data. Experimental results on some hyperspectral images demonstrate that the proposed SSPSR method enhances the details of the recovered high-resolution hyperspectral images.
  arXiv  Detail & Related papers  (2020-05-18T14:25:50Z)
• Hyperspectral-Multispectral Image Fusion with Weighted LASSO [68.04032419397677]
  We propose an approach for fusing hyperspectral and multispectral images to provide high-quality hyperspectral output. We demonstrate that the proposed sparse fusion and reconstruction provides quantitatively superior results when compared to existing methods on publicly available images.
  arXiv  Detail & Related papers  (2020-03-15T23:07:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.