CLTS-GAN: Color-Lighting-Texture-Specular Reflection Augmentation for Colonoscopy

• URL: http://arxiv.org/abs/2206.14951v1
• Date: Wed, 29 Jun 2022 23:51:16 GMT
• Title: CLTS-GAN: Color-Lighting-Texture-Specular Reflection Augmentation for Colonoscopy
• Authors: Shawn Mathew, Saad Nadeem, Arie Kaufman
• Abstract summary: CLTS-GAN is a new deep learning model that gives fine control over color, lighting, texture, and specular reflection for OC video frames. We show that adding colonoscopy-specific augmentations to the training data can improve state-of-the-art polyp detection/segmentation methods.
• Score: 5.298287413134345
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated analysis of optical colonoscopy (OC) video frames (to assist endoscopists during OC) is challenging due to variations in color, lighting, texture, and specular reflections. Previous methods either remove some of these variations via preprocessing (making pipelines cumbersome) or add diverse training data with annotations (but expensive and time-consuming). We present CLTS-GAN, a new deep learning model that gives fine control over color, lighting, texture, and specular reflection synthesis for OC video frames. We show that adding these colonoscopy-specific augmentations to the training data can improve state-of-the-art polyp detection/segmentation methods as well as drive next generation of OC simulators for training medical students. The code and pre-trained models for CLTS-GAN are available on Computational Endoscopy Platform GitHub (https://github.com/nadeemlab/CEP).

Related papers

• Gen-SIS: Generative Self-augmentation Improves Self-supervised Learning [52.170253590364545]
  Gen-SIS is a diffusion-based augmentation technique trained exclusively on unlabeled image data. We show that these self-augmentations', i.e. generative augmentations based on the vanilla SSL encoder embeddings, facilitate the training of a stronger SSL encoder.
  arXiv  Detail & Related papers  (2024-12-02T16:20:59Z)
• Leveraging Near-Field Lighting for Monocular Depth Estimation from Endoscopy Videos [12.497782583094281]
  Monocular depth estimation in endoscopy videos can enable assistive and robotic surgery to obtain better coverage of the organ and detection of various health issues. Despite promising progress on mainstream, natural image depth estimation, techniques perform poorly on endoscopy images. In this paper, we utilize the photometric cues, i.e., the light emitted from an endoscope and reflected by the surface, to improve monocular depth estimation.
  arXiv  Detail & Related papers  (2024-03-26T17:52:23Z)
• Endora: Video Generation Models as Endoscopy Simulators [53.72175969751398]
  This paper introduces model, an innovative approach to generate medical videos that simulate clinical endoscopy scenes. We also pioneer the first public benchmark for endoscopy simulation with video generation models. Endora marks a notable breakthrough in the deployment of generative AI for clinical endoscopy research.
  arXiv  Detail & Related papers  (2024-03-17T00:51:59Z)
• RT-GAN: Recurrent Temporal GAN for Adding Lightweight Temporal Consistency to Frame-Based Domain Translation Approaches [3.7873597471903944]
  We present a lightweight solution with a tunable temporal parameter, RT-GAN, for adding temporal consistency to individual frame-based approaches.<n>We demonstrate the effectiveness of our approach on two challenging use cases in colonoscopy.
  arXiv  Detail & Related papers  (2023-10-02T03:13:26Z)
• Improving Video Colorization by Test-Time Tuning [79.67548221384202]
  We propose an effective method, which aims to enhance video colorization through test-time tuning. By exploiting the reference to construct additional training samples during testing, our approach achieves a performance boost of 13 dB in PSNR on average.
  arXiv  Detail & Related papers  (2023-06-25T05:36:40Z)
• Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images [60.34381768479834]
  Recent advancements in diffusion models have enabled the generation of realistic deepfakes from textual prompts in natural language. We pioneer a systematic study on deepfake detection generated by state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-04-02T10:25:09Z)
• Automatic Generation of Synthetic Colonoscopy Videos for Domain Randomization [0.0]
  We propose an exemplary solution for synthesizing colonoscopy videos with substantial appearance and anatomical variations. This solution enables to learn discriminative domain-randomized representations of the interior colon while mimicking real-world settings.
  arXiv  Detail & Related papers  (2022-05-20T09:18:02Z)
• GAN Inversion for Data Augmentation to Improve Colonoscopy Lesion Classification [3.0100246737240877]
  We show that synthetic colonoscopy images generated by Generative Adversarial Network (GAN) inversion can be used as training data to improve the lesion classification performance of deep learning models. This approach inverts pairs of images with the same label to a semantically rich & disentangled latent space and manipulates latent representations to produce new synthetic images with the same label. We also generate realistic-looking synthetic lesion images by interpolating between original training images to increase the variety of lesion shapes in the training dataset.
  arXiv  Detail & Related papers  (2022-05-04T23:15:45Z)
• Colorectal Polyp Classification from White-light Colonoscopy Images via Domain Alignment [57.419727894848485]
  A computer-aided diagnosis system is required to assist accurate diagnosis from colonoscopy images. Most previous studies at-tempt to develop models for polyp differentiation using Narrow-Band Imaging (NBI) or other enhanced images. We propose a novel framework based on a teacher-student architecture for the accurate colorectal polyp classification.
  arXiv  Detail & Related papers  (2021-08-05T09:31:46Z)
• Ensembling with Deep Generative Views [72.70801582346344]
  generative models can synthesize "views" of artificial images that mimic real-world variations, such as changes in color or pose. Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification. We use StyleGAN2 as the source of generative augmentations and investigate this setup on classification tasks involving facial attributes, cat faces, and cars.
  arXiv  Detail & Related papers  (2021-04-29T17:58:35Z)
• Colonoscopy Polyp Detection: Domain Adaptation From Medical Report Images to Real-time Videos [76.37907640271806]
  We propose an Image-video-joint polyp detection network (Ivy-Net) to address the domain gap between colonoscopy images from historical medical reports and real-time videos. Experiments on the collected dataset demonstrate that our Ivy-Net achieves the state-of-the-art result on colonoscopy video.
  arXiv  Detail & Related papers  (2020-12-31T10:33:09Z)
• Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation [5.861206243996454]
  We present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between optical colonoscopy (OC) and virtual colonoscopy (VC) We show results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models.
  arXiv  Detail & Related papers  (2020-03-27T15:34:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.