A self-supervised and adversarial approach to hyperspectral demosaicking and RGB reconstruction in surgical imaging

• URL: http://arxiv.org/abs/2407.19282v1
• Date: Sat, 27 Jul 2024 15:29:35 GMT
• Title: A self-supervised and adversarial approach to hyperspectral demosaicking and RGB reconstruction in surgical imaging
• Authors: Peichao Li, Oscar MacCormac, Jonathan Shapey, Tom Vercauteren,
• Abstract summary: Hyperspectral imaging holds promises in surgical imaging by offering biological tissue differentiation capabilities with detailed information that is invisible to the naked eye. For intra-operative guidance, real-time spectral data capture and display is mandated. Snapshot mosaic hyperspectral cameras are currently seen as the most suitable technology given this requirement. We present a self-supervised demosaicking and RGB reconstruction method that does not depend on paired high-resolution data as ground truth.
• Score: 3.426432165500852
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Hyperspectral imaging holds promises in surgical imaging by offering biological tissue differentiation capabilities with detailed information that is invisible to the naked eye. For intra-operative guidance, real-time spectral data capture and display is mandated. Snapshot mosaic hyperspectral cameras are currently seen as the most suitable technology given this requirement. However, snapshot mosaic imaging requires a demosaicking algorithm to fully restore the spatial and spectral details in the images. Modern demosaicking approaches typically rely on synthetic datasets to develop supervised learning methods, as it is practically impossible to simultaneously capture both snapshot and high-resolution spectral images of the exact same surgical scene. In this work, we present a self-supervised demosaicking and RGB reconstruction method that does not depend on paired high-resolution data as ground truth. We leverage unpaired standard high-resolution surgical microscopy images, which only provide RGB data but can be collected during routine surgeries. Adversarial learning complemented by self-supervised approaches are used to drive our hyperspectral-based RGB reconstruction into resembling surgical microscopy images and increasing the spatial resolution of our demosaicking. The spatial and spectral fidelity of the reconstructed hyperspectral images have been evaluated quantitatively. Moreover, a user study was conducted to evaluate the RGB visualisation generated from these spectral images. Both spatial detail and colour accuracy were assessed by neurosurgical experts. Our proposed self-supervised demosaicking method demonstrates improved results compared to existing methods, demonstrating its potential for seamless integration into intra-operative workflows.

Related papers

• RGB to Hyperspectral: Spectral Reconstruction for Enhanced Surgical Imaging [7.2993064695496255]
  This study investigates the reconstruction of hyperspectral signatures from RGB data to enhance surgical imaging. Various architectures based on convolutional neural networks (CNNs) and transformer models are evaluated. Transformer models exhibit superior performance in terms of RMSE, SAM, PSNR and SSIM.
  arXiv  Detail & Related papers  (2024-10-17T14:05:41Z)
• A Unified Model for Compressed Sensing MRI Across Undersampling Patterns [69.19631302047569]
  Deep neural networks have shown great potential for reconstructing high-fidelity images from undersampled measurements. Our model is based on neural operators, a discretization-agnostic architecture. Our inference speed is also 1,400x faster than diffusion methods.
  arXiv  Detail & Related papers  (2024-10-05T20:03:57Z)
• On Sensitivity and Robustness of Normalization Schemes to Input Distribution Shifts in Automatic MR Image Diagnosis [58.634791552376235]
  Deep Learning (DL) models have achieved state-of-the-art performance in diagnosing multiple diseases using reconstructed images as input. DL models are sensitive to varying artifacts as it leads to changes in the input data distribution between the training and testing phases. We propose to use other normalization techniques, such as Group Normalization and Layer Normalization, to inject robustness into model performance against varying image artifacts.
  arXiv  Detail & Related papers  (2023-06-23T03:09:03Z)
• mHealth hyperspectral learning for instantaneous spatiospectral imaging of hemodynamics [0.2638512174804417]
  Hyperspectral learning exploits idea that a photograph is more than merely a picture and contains detailed spectral information. A small sampling of hyperspectral data enables spectrally informed learning to recover a hypercube from an RGB image. Hyperspectral learning is capable of recovering full spectroscopic resolution in the hypercube, comparable to high spectral resolutions of scientific spectrometers.
  arXiv  Detail & Related papers  (2023-03-27T15:12:10Z)
• Spatial gradient consistency for unsupervised learning of hyperspectral demosaicking: Application to surgical imaging [4.795951381086172]
  Hyperspectral imaging has the potential to improve tissue characterisation in real-time and with high-resolution. A demosaicking algorithm is required to fully recover the spatial and spectral information of the snapshot images. We present a fully unsupervised hyperspectral image demosaicking algorithm which only requires snapshot images for training purposes.
  arXiv  Detail & Related papers  (2023-02-21T18:07:14Z)
• OADAT: Experimental and Synthetic Clinical Optoacoustic Data for Standardized Image Processing [62.993663757843464]
  Optoacoustic (OA) imaging is based on excitation of biological tissues with nanosecond-duration laser pulses followed by detection of ultrasound waves generated via light-absorption-mediated thermoelastic expansion. OA imaging features a powerful combination between rich optical contrast and high resolution in deep tissues. No standardized datasets generated with different types of experimental set-up and associated processing methods are available to facilitate advances in broader applications of OA in clinical settings.
  arXiv  Detail & Related papers  (2022-06-17T08:11:26Z)
• A Temporal Learning Approach to Inpainting Endoscopic Specularities and Its effect on Image Correspondence [13.25903945009516]
  We propose using a temporal generative adversarial network (GAN) to inpaint the hidden anatomy under specularities. This is achieved using in-vivo data of gastric endoscopy (Hyper-Kvasir) in a fully unsupervised manner. We also assess the effect of our method in computer vision tasks that underpin 3D reconstruction and camera motion estimation.
  arXiv  Detail & Related papers  (2022-03-31T13:14:00Z)
• Multi-modal Aggregation Network for Fast MR Imaging [85.25000133194762]
  We propose a novel Multi-modal Aggregation Network, named MANet, which is capable of discovering complementary representations from a fully sampled auxiliary modality. In our MANet, the representations from the fully sampled auxiliary and undersampled target modalities are learned independently through a specific network. Our MANet follows a hybrid domain learning framework, which allows it to simultaneously recover the frequency signal in the $k$-space domain.
  arXiv  Detail & Related papers  (2021-10-15T13:16:59Z)
• Deep Learning Approach for Hyperspectral Image Demosaicking, Spectral Correction and High-resolution RGB Reconstruction [3.0478210530038443]
  We propose a deep learning-based image demosaicking algorithm for snapshot hyperspectral images using supervised learning methods. Due to the lack of publicly available medical images, a synthetic image generation approach is proposed to simulate snapshot images from existing medical image datasets. The resulting demosaicked images are evaluated both quantitatively and qualitatively, showing clear improvements in image quality.
  arXiv  Detail & Related papers  (2021-09-03T09:50:03Z)
• Spectral-Spatial Recurrent-Convolutional Networks for In-Vivo Hyperspectral Tumor Type Classification [49.32653090178743]
  We demonstrate the feasibility of in-vivo tumor type classification using hyperspectral imaging and deep learning. Our best model achieves an AUC of 76.3%, significantly outperforming previous conventional and deep learning methods.
  arXiv  Detail & Related papers  (2020-07-02T12:00:53Z)
• Hyperspectral-Multispectral Image Fusion with Weighted LASSO [68.04032419397677]
  We propose an approach for fusing hyperspectral and multispectral images to provide high-quality hyperspectral output. We demonstrate that the proposed sparse fusion and reconstruction provides quantitatively superior results when compared to existing methods on publicly available images.
  arXiv  Detail & Related papers  (2020-03-15T23:07:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.