Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: Application to surgical imaging

• URL: http://arxiv.org/abs/2302.10927v1
• Date: Tue, 21 Feb 2023 18:07:14 GMT
• Title: Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: Application to surgical imaging
• Authors: Peichao Li, Muhammad Asad, Conor Horgan, Oscar MacCormac, Jonathan Shapey, Tom Vercauteren
• Abstract summary: Hyperspectral imaging has the potential to improve tissue characterisation in real-time and with high-resolution. A demosaicking algorithm is required to fully recover the spatial and spectral information of the snapshot images. We present a fully unsupervised hyperspectral image demosaicking algorithm which only requires snapshot images for training purposes.
• Score: 4.795951381086172
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyperspectral imaging has the potential to improve intraoperative decision making if tissue characterisation is performed in real-time and with high-resolution. Hyperspectral snapshot mosaic sensors offer a promising approach due to their fast acquisition speed and compact size. However, a demosaicking algorithm is required to fully recover the spatial and spectral information of the snapshot images. Most state-of-the-art demosaicking algorithms require ground-truth training data with paired snapshot and high-resolution hyperspectral images, but such imagery pairs with the exact same scene are physically impossible to acquire in intraoperative settings. In this work, we present a fully unsupervised hyperspectral image demosaicking algorithm which only requires exemplar snapshot images for training purposes. We regard hyperspectral demosaicking as an ill-posed linear inverse problem which we solve using a deep neural network. We take advantage of the spectral correlation occurring in natural scenes to design a novel inter spectral band regularisation term based on spatial gradient consistency. By combining our proposed term with standard regularisation techniques and exploiting a standard data fidelity term, we obtain an unsupervised loss function for training deep neural networks, which allows us to achieve real-time hyperspectral image demosaicking. Quantitative results on hyperspetral image datasets show that our unsupervised demosaicking approach can achieve similar performance to its supervised counter-part, and significantly outperform linear demosaicking. A qualitative user study on real snapshot hyperspectral surgical images confirms the results from the quantitative analysis. Our results suggest that the proposed unsupervised algorithm can achieve promising hyperspectral demosaicking in real-time thus advancing the suitability of the modality for intraoperative use.

Related papers

• A self-supervised and adversarial approach to hyperspectral demosaicking and RGB reconstruction in surgical imaging [3.426432165500852]
  Hyperspectral imaging holds promises in surgical imaging by offering biological tissue differentiation capabilities with detailed information that is invisible to the naked eye. For intra-operative guidance, real-time spectral data capture and display is mandated. Snapshot mosaic hyperspectral cameras are currently seen as the most suitable technology given this requirement. We present a self-supervised demosaicking and RGB reconstruction method that does not depend on paired high-resolution data as ground truth.
  arXiv  Detail & Related papers  (2024-07-27T15:29:35Z)
• MMNet: Multi-Collaboration and Multi-Supervision Network for Sequential Deepfake Detection [81.59191603867586]
  Sequential deepfake detection aims to identify forged facial regions with the correct sequence for recovery. The recovery of forged images requires knowledge of the manipulation model to implement inverse transformations. We propose Multi-Collaboration and Multi-Supervision Network (MMNet) that handles various spatial scales and sequential permutations in forged face images.
  arXiv  Detail & Related papers  (2023-07-06T02:32:08Z)
• Self-Supervised Endoscopic Image Key-Points Matching [1.3764085113103222]
  This paper proposes a novel self-supervised approach for endoscopic image matching based on deep learning techniques. Our method outperformed standard hand-crafted local feature descriptors in terms of precision and recall.
  arXiv  Detail & Related papers  (2022-08-24T10:47:21Z)
• Learning Discriminative Shrinkage Deep Networks for Image Deconvolution [122.79108159874426]
  We propose an effective non-blind deconvolution approach by learning discriminative shrinkage functions to implicitly model these terms. Experimental results show that the proposed method performs favorably against the state-of-the-art ones in terms of efficiency and accuracy.
  arXiv  Detail & Related papers  (2021-11-27T12:12:57Z)
• Sharp-GAN: Sharpness Loss Regularized GAN for Histopathology Image Synthesis [65.47507533905188]
  Conditional generative adversarial networks have been applied to generate synthetic histopathology images. We propose a sharpness loss regularized generative adversarial network to synthesize realistic histopathology images.
  arXiv  Detail & Related papers  (2021-10-27T18:54:25Z)
• Deep Learning Approach for Hyperspectral Image Demosaicking, Spectral Correction and High-resolution RGB Reconstruction [3.0478210530038443]
  We propose a deep learning-based image demosaicking algorithm for snapshot hyperspectral images using supervised learning methods. Due to the lack of publicly available medical images, a synthetic image generation approach is proposed to simulate snapshot images from existing medical image datasets. The resulting demosaicked images are evaluated both quantitatively and qualitatively, showing clear improvements in image quality.
  arXiv  Detail & Related papers  (2021-09-03T09:50:03Z)
• STRESS: Super-Resolution for Dynamic Fetal MRI using Self-Supervised Learning [2.5581619987137048]
  We propose STRESS, a self-supervised super-resolution framework for dynamic fetal MRI with interleaved slice acquisitions. Our proposed method simulates an interleaved slice acquisition along the high-resolution axis on the originally acquired data to generate pairs of low- and high-resolution images. Evaluations on both simulated and in utero data show that our proposed method outperforms other self-supervised super-resolution methods.
  arXiv  Detail & Related papers  (2021-06-23T13:52:11Z)
• Unsupervised Spatial-spectral Network Learning for Hyperspectral Compressive Snapshot Reconstruction [16.530040002441694]
  We propose an unsupervised spatial-spectral network to reconstruct hyperspectral images only from the compressive snapshot measurement. Our network can achieve better reconstruction results than the state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-12-18T12:29:04Z)
• Improved Slice-wise Tumour Detection in Brain MRIs by Computing Dissimilarities between Latent Representations [68.8204255655161]
  Anomaly detection for Magnetic Resonance Images (MRIs) can be solved with unsupervised methods. We have proposed a slice-wise semi-supervised method for tumour detection based on the computation of a dissimilarity function in the latent space of a Variational AutoEncoder. We show that by training the models on higher resolution images and by improving the quality of the reconstructions, we obtain results which are comparable with different baselines.
  arXiv  Detail & Related papers  (2020-07-24T14:02:09Z)
• Learning Spatial-Spectral Prior for Super-Resolution of Hyperspectral Imagery [79.69449412334188]
  In this paper, we investigate how to adapt state-of-the-art residual learning based single gray/RGB image super-resolution approaches. We introduce a spatial-spectral prior network (SSPN) to fully exploit the spatial information and the correlation between the spectra of the hyperspectral data. Experimental results on some hyperspectral images demonstrate that the proposed SSPSR method enhances the details of the recovered high-resolution hyperspectral images.
  arXiv  Detail & Related papers  (2020-05-18T14:25:50Z)
• DeFeat-Net: General Monocular Depth via Simultaneous Unsupervised Representation Learning [65.94499390875046]
  DeFeat-Net is an approach to simultaneously learn a cross-domain dense feature representation. Our technique is able to outperform the current state-of-the-art with around 10% reduction in all error measures.
  arXiv  Detail & Related papers  (2020-03-30T13:10:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.