Generalizing Surgical Instruments Segmentation to Unseen Domains with One-to-Many Synthesis

• URL: http://arxiv.org/abs/2306.16285v1
• Date: Wed, 28 Jun 2023 15:06:44 GMT
• Title: Generalizing Surgical Instruments Segmentation to Unseen Domains with One-to-Many Synthesis
• Authors: An Wang, Mobarakol Islam, Mengya Xu, Hongliang Ren
• Abstract summary: Deep learning methods are frequently hindered from deploying to real-world surgical applications. Data collection, annotation, and domain shift in-between sites and patients are the most common obstacles. We mitigate data-related issues by efficiently leveraging minimal source images to generate synthetic surgical instrument segmentation datasets.
• Score: 18.830738606514736
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite their impressive performance in various surgical scene understanding tasks, deep learning-based methods are frequently hindered from deploying to real-world surgical applications for various causes. Particularly, data collection, annotation, and domain shift in-between sites and patients are the most common obstacles. In this work, we mitigate data-related issues by efficiently leveraging minimal source images to generate synthetic surgical instrument segmentation datasets and achieve outstanding generalization performance on unseen real domains. Specifically, in our framework, only one background tissue image and at most three images of each foreground instrument are taken as the seed images. These source images are extensively transformed and employed to build up the foreground and background image pools, from which randomly sampled tissue and instrument images are composed with multiple blending techniques to generate new surgical scene images. Besides, we introduce hybrid training-time augmentations to diversify the training data further. Extensive evaluation on three real-world datasets, i.e., Endo2017, Endo2018, and RoboTool, demonstrates that our one-to-many synthetic surgical instruments datasets generation and segmentation framework can achieve encouraging performance compared with training with real data. Notably, on the RoboTool dataset, where a more significant domain gap exists, our framework shows its superiority of generalization by a considerable margin. We expect that our inspiring results will attract research attention to improving model generalization with data synthesizing.

Related papers

• UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation [64.01742988773745]
  An increasing privacy concern exists regarding training large-scale image segmentation models on unauthorized private data. We exploit the concept of unlearnable examples to make images unusable to model training by generating and adding unlearnable noise into the original images. We empirically verify the effectiveness of UnSeg across 6 mainstream image segmentation tasks, 10 widely used datasets, and 7 different network architectures.
  arXiv  Detail & Related papers  (2024-10-13T16:34:46Z)
• Data Augmentation for Surgical Scene Segmentation with Anatomy-Aware Diffusion Models [1.9085155846692308]
  We introduce a multi-stage approach to generate multi-class surgical datasets with annotations. Our framework improves anatomy awareness by training organ specific models with an inpainting objective guided by binary segmentation masks. This versatile approach allows the generation of multi-class datasets from real binary datasets and simulated surgical masks.
  arXiv  Detail & Related papers  (2024-10-10T09:29:23Z)
• Discriminative Hamiltonian Variational Autoencoder for Accurate Tumor Segmentation in Data-Scarce Regimes [2.8498944632323755]
  We propose an end-to-end hybrid architecture for medical image segmentation. We use Hamiltonian Variational Autoencoders (HVAE) and a discriminative regularization to improve the quality of generated images. Our architecture operates on a slice-by-slice basis to segment 3D volumes, capitilizing on the richly augmented dataset.
  arXiv  Detail & Related papers  (2024-06-17T15:42:08Z)
• SatSynth: Augmenting Image-Mask Pairs through Diffusion Models for Aerial Semantic Segmentation [69.42764583465508]
  We explore the potential of generative image diffusion to address the scarcity of annotated data in earth observation tasks. To the best of our knowledge, we are the first to generate both images and corresponding masks for satellite segmentation.
  arXiv  Detail & Related papers  (2024-03-25T10:30:22Z)
• Deep Domain Adaptation: A Sim2Real Neural Approach for Improving Eye-Tracking Systems [80.62854148838359]
  Eye image segmentation is a critical step in eye tracking that has great influence over the final gaze estimate. We use dimensionality-reduction techniques to measure the overlap between the target eye images and synthetic training data. Our methods result in robust, improved performance when tackling the discrepancy between simulation and real-world data samples.
  arXiv  Detail & Related papers  (2024-03-23T22:32:06Z)
• AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images [53.29794593104923]
  We present a novel concept of shared-context processing for whole slide histopathology images. AMIGO uses the celluar graph within the tissue to provide a single representation for a patient. We show that our model is strongly robust to missing information to an extent that it can achieve the same performance with as low as 20% of the data.
  arXiv  Detail & Related papers  (2023-03-01T23:37:45Z)
• Rethinking Surgical Instrument Segmentation: A Background Image Can Be All You Need [18.830738606514736]
  Data scarcity and imbalance have heavily affected the model accuracy and limited the design and deployment of deep learning-based surgical applications. We propose a one-to-many data generation solution that gets rid of the complicated and expensive process of data collection and annotation from robotic surgery. Our empirical analysis suggests that without the high cost of data collection and annotation, we can achieve decent surgical instrument segmentation performance.
  arXiv  Detail & Related papers  (2022-06-23T16:22:56Z)
• Reducing Annotating Load: Active Learning with Synthetic Images in Surgical Instrument Segmentation [11.705954708866079]
  instrument segmentation in endoscopic vision of robot-assisted surgery is challenging due to reflection on the instruments and frequent contacts with tissue. Deep neural networks (DNN) show competitive performance and are in favor in recent years. Motivated by alleviating this workload, we propose a general embeddable method to decrease the usage of labeled real images.
  arXiv  Detail & Related papers  (2021-08-07T22:30:53Z)
• Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization [112.68171734288237]
  We propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. We learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of unlabeled images. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization.
  arXiv  Detail & Related papers  (2021-04-12T21:41:25Z)
• Multi-Spectral Image Synthesis for Crop/Weed Segmentation in Precision Farming [3.4788711710826083]
  We propose an alternative solution with respect to the common data augmentation methods, applying it to the problem of crop/weed segmentation in precision farming. We create semi-artificial samples by replacing the most relevant object classes (i.e., crop and weeds) with their synthesized counterparts. In addition to RGB data, we take into account also near-infrared (NIR) information, generating four channel multi-spectral synthetic images.
  arXiv  Detail & Related papers  (2020-09-12T08:49:36Z)
• Pathological Retinal Region Segmentation From OCT Images Using Geometric Relation Based Augmentation [84.7571086566595]
  We propose improvements over previous GAN-based medical image synthesis methods by jointly encoding the intrinsic relationship of geometry and shape. The proposed method outperforms state-of-the-art segmentation methods on the public RETOUCH dataset having images captured from different acquisition procedures.
  arXiv  Detail & Related papers  (2020-03-31T11:50:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.