Semantic segmentation of surgical hyperspectral images under geometric domain shifts

• URL: http://arxiv.org/abs/2303.10972v2
• Date: Mon, 18 Sep 2023 01:31:14 GMT
• Title: Semantic segmentation of surgical hyperspectral images under geometric domain shifts
• Authors: Jan Sellner and Silvia Seidlitz, Alexander Studier-Fischer, Alessandro Motta, Berkin \"Ozdemir, Beat Peter M\"uller-Stich, Felix Nickel, Lena Maier-Hein
• Abstract summary: We present the first analysis of state-of-the-art semantic segmentation networks in the presence of geometric out-of-distribution (OOD) data. We also address generalizability with a dedicated augmentation technique termed "Organ Transplantation" Our scheme improves on the SOA DSC by up to 67 % (RGB) and 90 % (HSI) and renders performance on par with in-distribution performance on real OOD test data.
• Score: 69.91792194237212
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robust semantic segmentation of intraoperative image data could pave the way for automatic surgical scene understanding and autonomous robotic surgery. Geometric domain shifts, however, although common in real-world open surgeries due to variations in surgical procedures or situs occlusions, remain a topic largely unaddressed in the field. To address this gap in the literature, we (1) present the first analysis of state-of-the-art (SOA) semantic segmentation networks in the presence of geometric out-of-distribution (OOD) data, and (2) address generalizability with a dedicated augmentation technique termed "Organ Transplantation" that we adapted from the general computer vision community. According to a comprehensive validation on six different OOD data sets comprising 600 RGB and hyperspectral imaging (HSI) cubes from 33 pigs semantically annotated with 19 classes, we demonstrate a large performance drop of SOA organ segmentation networks applied to geometric OOD data. Surprisingly, this holds true not only for conventional RGB data (drop of Dice similarity coefficient (DSC) by 46 %) but also for HSI data (drop by 45 %), despite the latter's rich information content per pixel. Using our augmentation scheme improves on the SOA DSC by up to 67 % (RGB) and 90 % (HSI) and renders performance on par with in-distribution performance on real OOD test data. The simplicity and effectiveness of our augmentation scheme makes it a valuable network-independent tool for addressing geometric domain shifts in semantic scene segmentation of intraoperative data. Our code and pre-trained models are available at https://github.com/IMSY-DKFZ/htc.

Related papers

• Handling Geometric Domain Shifts in Semantic Segmentation of Surgical RGB and Hyperspectral Images [67.66644395272075]
  We present first analysis of state-of-the-art semantic segmentation models when faced with geometric out-of-distribution data. We propose an augmentation technique called "Organ Transplantation" to enhance generalizability. Our augmentation technique improves SOA model performance by up to 67 % for RGB data and 90 % for HSI data, achieving performance at the level of in-distribution performance on real OOD test data.
  arXiv  Detail & Related papers  (2024-08-27T19:13:15Z)
• Self-Supervised Correction Learning for Semi-Supervised Biomedical Image Segmentation [84.58210297703714]
  We propose a self-supervised correction learning paradigm for semi-supervised biomedical image segmentation. We design a dual-task network, including a shared encoder and two independent decoders for segmentation and lesion region inpainting. Experiments on three medical image segmentation datasets for different tasks demonstrate the outstanding performance of our method.
  arXiv  Detail & Related papers  (2023-01-12T08:19:46Z)
• SD-LayerNet: Semi-supervised retinal layer segmentation in OCT using disentangled representation with anatomical priors [4.2663199451998475]
  We introduce a semi-supervised paradigm into the retinal layer segmentation task. In particular, a novel fully differentiable approach is used for converting surface position regression into a pixel-wise structured segmentation. In parallel, we propose a set of anatomical priors to improve network training when a limited amount of labeled data is available.
  arXiv  Detail & Related papers  (2022-07-01T14:30:59Z)
• Two-Stream Graph Convolutional Network for Intra-oral Scanner Image Segmentation [133.02190910009384]
  We propose a two-stream graph convolutional network (i.e., TSGCN) to handle inter-view confusion between different raw attributes. Our TSGCN significantly outperforms state-of-the-art methods in 3D tooth (surface) segmentation.
  arXiv  Detail & Related papers  (2022-04-19T10:41:09Z)
• 4D-OR: Semantic Scene Graphs for OR Domain Modeling [72.1320671045942]
  We propose using semantic scene graphs (SSG) to describe and summarize the surgical scene. The nodes of the scene graphs represent different actors and objects in the room, such as medical staff, patients, and medical equipment. We create the first publicly available 4D surgical SSG dataset, 4D-OR, containing ten simulated total knee replacement surgeries.
  arXiv  Detail & Related papers  (2022-03-22T17:59:45Z)
• Robust deep learning-based semantic organ segmentation in hyperspectral images [29.342448910787773]
  Full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. We are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting. We conclude that HSI could become a powerful image modality for fully-automatic surgical scene understanding.
  arXiv  Detail & Related papers  (2021-11-09T20:37:38Z)
• Towards a Computed-Aided Diagnosis System in Colonoscopy: Automatic Polyp Segmentation Using Convolution Neural Networks [10.930181796935734]
  We present a deep learning framework for recognizing lesions in colonoscopy and capsule endoscopy images. To our knowledge, we present the first work to use FCNs for polyp segmentation in addition to proposing a novel combination of SfS and RGB that boosts performance.
  arXiv  Detail & Related papers  (2021-01-15T10:08:53Z)
• Pathological Retinal Region Segmentation From OCT Images Using Geometric Relation Based Augmentation [84.7571086566595]
  We propose improvements over previous GAN-based medical image synthesis methods by jointly encoding the intrinsic relationship of geometry and shape. The proposed method outperforms state-of-the-art segmentation methods on the public RETOUCH dataset having images captured from different acquisition procedures.
  arXiv  Detail & Related papers  (2020-03-31T11:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.