Robust deep learning-based semantic organ segmentation in hyperspectral images

• URL: http://arxiv.org/abs/2111.05408v1
• Date: Tue, 9 Nov 2021 20:37:38 GMT
• Title: Robust deep learning-based semantic organ segmentation in hyperspectral images
• Authors: Silvia Seidlitz (1 and 2) and Jan Sellner (1 and 2), Jan Odenthal (3), Berkin \"Ozdemir (3 and 4), Alexander Studier-Fischer (3 and 4), Samuel Kn\"odler (3 and 4), Leonardo Ayala (1 and 4), Tim Adler (1 and 6), Hannes G. Kenngott (2 and 3), Minu Tizabi (1), Martin Wagner (2 and 3 and 4), Felix Nickel (2 and 3 and 4), Beat P. M\"uller-Stich (3 and 4), Lena Maier-Hein (1 and 2 and 4 and 5 and 6) ((1) Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), Heidelberg, Germany, (2) Helmholtz Information and Data Science School for Health, Karlsruhe/Heidelberg, Germany, (3) Department of General, Visceral, and Transplantation Surgery, Heidelberg University Hospital, Heidelberg, Germany, (4) Medical Faculty, Heidelberg University, Heidelberg, Germany, (5) HIP Helmholtz Imaging Platform, German Cancer Research Center (DKFZ), Heidelberg, Germany, (6) Faculty of Mathematics and Computer Science, Heidelberg University, Germany)
• Abstract summary: Full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. We are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting. We conclude that HSI could become a powerful image modality for fully-automatic surgical scene understanding.
• Score: 29.342448910787773
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Semantic image segmentation is an important prerequisite for context-awareness and autonomous robotics in surgery. The state of the art has focused on conventional RGB video data acquired during minimally invasive surgery, but full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. To address this gap in the literature, we are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting: (1) What is an adequate representation of HSI data for neural network-based fully automated organ segmentation, especially with respect to the spatial granularity of the data (pixels vs. superpixels vs. patches vs. full images)? (2) Is there a benefit of using HSI data compared to other modalities, namely RGB data and processed HSI data (e.g. tissue parameters like oxygenation), when performing semantic organ segmentation? According to a comprehensive validation study based on 506 HSI images from 20 pigs, annotated with a total of 19 classes, deep learning-based segmentation performance increases - consistently across modalities - with the spatial context of the input data. Unprocessed HSI data offers an advantage over RGB data or processed data from the camera provider, with the advantage increasing with decreasing size of the input to the neural network. Maximum performance (HSI applied to whole images) yielded a mean dice similarity coefficient (DSC) of 0.89 (standard deviation (SD) 0.04), which is in the range of the inter-rater variability (DSC of 0.89 (SD 0.07)). We conclude that HSI could become a powerful image modality for fully-automatic surgical scene understanding with many advantages over traditional imaging, including the ability to recover additional functional tissue information.

Related papers

• Handling Geometric Domain Shifts in Semantic Segmentation of Surgical RGB and Hyperspectral Images [67.66644395272075]
  We present first analysis of state-of-the-art semantic segmentation models when faced with geometric out-of-distribution data. We propose an augmentation technique called "Organ Transplantation" to enhance generalizability. Our augmentation technique improves SOA model performance by up to 67 % for RGB data and 90 % for HSI data, achieving performance at the level of in-distribution performance on real OOD test data.
  arXiv  Detail & Related papers  (2024-08-27T19:13:15Z)
• TotalSegmentator MRI: Sequence-Independent Segmentation of 59 Anatomical Structures in MR images [62.53931644063323]
  In this study we extended the capabilities of TotalSegmentator to MR images. We trained an nnU-Net segmentation algorithm on this dataset and calculated similarity coefficients (Dice) to evaluate the model's performance. The model significantly outperformed two other publicly available segmentation models (Dice score 0.824 versus 0.762; p0.001 and 0.762 versus 0.542; p)
  arXiv  Detail & Related papers  (2024-05-29T20:15:54Z)
• Robust Tumor Segmentation with Hyperspectral Imaging and Graph Neural Networks [31.87960207119459]
  We propose an improved methodology that leverages the spatial context of tiles for more robust and smoother segmentation. To address the irregular shapes of tiles, we utilize Graph Neural Networks (GNNs) to propagate context information across neighboring regions. Our findings demonstrate that context-aware GNN algorithms can robustly find tumor demarcations on HSI images.
  arXiv  Detail & Related papers  (2023-11-20T14:07:38Z)
• Semantic segmentation of surgical hyperspectral images under geometric domain shifts [69.91792194237212]
  We present the first analysis of state-of-the-art semantic segmentation networks in the presence of geometric out-of-distribution (OOD) data. We also address generalizability with a dedicated augmentation technique termed "Organ Transplantation" Our scheme improves on the SOA DSC by up to 67 % (RGB) and 90 % (HSI) and renders performance on par with in-distribution performance on real OOD test data.
  arXiv  Detail & Related papers  (2023-03-20T09:50:07Z)
• Hyperspectral Image Segmentation: A Preliminary Study on the Oral and Dental Spectral Image Database (ODSI-DB) [3.1993890965689666]
  Hyperspectral imaging (HSI) is a promising technology providing rich spectral information. Recent development of snapshot HSI cameras has made real-time clinical HSI a distinct possibility. Our work highlights the relative importance of spectral resolution, spectral range, and spatial information to both guide the development of HSI cameras and inform future clinical HSI applications.
  arXiv  Detail & Related papers  (2023-03-14T21:57:11Z)
• Two-Stream Graph Convolutional Network for Intra-oral Scanner Image Segmentation [133.02190910009384]
  We propose a two-stream graph convolutional network (i.e., TSGCN) to handle inter-view confusion between different raw attributes. Our TSGCN significantly outperforms state-of-the-art methods in 3D tooth (surface) segmentation.
  arXiv  Detail & Related papers  (2022-04-19T10:41:09Z)
• Pathological Retinal Region Segmentation From OCT Images Using Geometric Relation Based Augmentation [84.7571086566595]
  We propose improvements over previous GAN-based medical image synthesis methods by jointly encoding the intrinsic relationship of geometry and shape. The proposed method outperforms state-of-the-art segmentation methods on the public RETOUCH dataset having images captured from different acquisition procedures.
  arXiv  Detail & Related papers  (2020-03-31T11:50:43Z)
• Gastric histopathology image segmentation using a hierarchical conditional random field [16.920864110707747]
  A novel Conditional Random Field (HCRF) based Gastric Histopathology Image (GHIS) method is proposed. Our HCRF model demonstrates high segmentation performance and shows its effectiveness and future potential in the GHIS field.
  arXiv  Detail & Related papers  (2020-03-03T02:44:31Z)
• VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images [121.31355003451152]
  Large Scale Vertebrae Challenge (VerSe) was organised in conjunction with the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) in 2019 and 2020. We present the the results of this evaluation and further investigate the performance-variation at vertebra-level, scan-level, and at different fields-of-view.
  arXiv  Detail & Related papers  (2020-01-24T21:09:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.