Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation

• URL: http://arxiv.org/abs/2003.12473v3
• Date: Thu, 26 Aug 2021 09:31:33 GMT
• Title: Augmenting Colonoscopy using Extended and Directional CycleGAN for Lossy Image Translation
• Authors: Shawn Mathew, Saad Nadeem, Sruti Kumari, Arie Kaufman
• Abstract summary: We present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between optical colonoscopy (OC) and virtual colonoscopy (VC) We show results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models.
• Score: 5.861206243996454
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Colorectal cancer screening modalities, such as optical colonoscopy (OC) and virtual colonoscopy (VC), are critical for diagnosing and ultimately removing polyps (precursors of colon cancer). The non-invasive VC is normally used to inspect a 3D reconstructed colon (from CT scans) for polyps and if found, the OC procedure is performed to physically traverse the colon via endoscope and remove these polyps. In this paper, we present a deep learning framework, Extended and Directional CycleGAN, for lossy unpaired image-to-image translation between OC and VC to augment OC video sequences with scale-consistent depth information from VC, and augment VC with patient-specific textures, color and specular highlights from OC (e.g, for realistic polyp synthesis). Both OC and VC contain structural information, but it is obscured in OC by additional patient-specific texture and specular highlights, hence making the translation from OC to VC lossy. The existing CycleGAN approaches do not handle lossy transformations. To address this shortcoming, we introduce an extended cycle consistency loss, which compares the geometric structures from OC in the VC domain. This loss removes the need for the CycleGAN to embed OC information in the VC domain. To handle a stronger removal of the textures and lighting, a Directional Discriminator is introduced to differentiate the direction of translation (by creating paired information for the discriminator), as opposed to the standard CycleGAN which is direction-agnostic. Combining the extended cycle consistency loss and the Directional Discriminator, we show state-of-the-art results on scale-consistent depth inference for phantom, textured VC and for real polyp and normal colon video sequences. We also present results for realistic pendunculated and flat polyp synthesis from bumps introduced in 3D VC models. Code/models: https://github.com/nadeemlab/CEP.

Related papers

• EndoFinder: Online Lesion Retrieval for Explainable Colorectal Polyp Diagnosis Leveraging Latent Scene Representations [10.83700068295662]
  Colorectal cancer (CRC) remains a leading cause of cancer-related mortality, underscoring the importance of timely polyp detection and diagnosis.<n>We propose EndoFinder, an online polyp retrieval framework that leverages multi-view scene representations for explainable and scalable CRC diagnosis.
  arXiv  Detail & Related papers  (2025-07-23T08:45:19Z)
• Lesion-aware Dynamic Kernel for Polyp Segmentation [49.63274623103663]
  We propose a lesion-aware dynamic network (LDNet) for polyp segmentation. It is a traditional u-shape encoder-decoder structure incorporated with a dynamic kernel generation and updating scheme. This simple but effective scheme endows our model with powerful segmentation performance and generalization capability.
  arXiv  Detail & Related papers  (2023-01-12T09:53:57Z)
• DilatedSegNet: A Deep Dilated Segmentation Network for Polyp Segmentation [2.6179759969345002]
  Colorectal cancer (CRC) is the second leading cause of cancer-related death worldwide. Powered by deep learning, computer-aided diagnosis (CAD) systems can detect regions in the colon overlooked by physicians during colonoscopy. Lacking high accuracy and real-time speed are the essential obstacles to be overcome for successful clinical integration of such systems.
  arXiv  Detail & Related papers  (2022-10-24T20:36:30Z)
• REGAS: REspiratory-GAted Synthesis of Views for Multi-Phase CBCT Reconstruction from a single 3D CBCT Acquisition [75.64791080418162]
  REGAS proposes a self-supervised method to synthesize the undersampled tomographic views and mitigate aliasing artifacts in reconstructed images. To address the large memory cost of deep neural networks on high resolution 4D data, REGAS introduces a novel Ray Path Transformation (RPT) that allows for distributed, differentiable forward projections.
  arXiv  Detail & Related papers  (2022-08-17T03:42:19Z)
• CLTS-GAN: Color-Lighting-Texture-Specular Reflection Augmentation for Colonoscopy [5.298287413134345]
  CLTS-GAN is a new deep learning model that gives fine control over color, lighting, texture, and specular reflection for OC video frames. We show that adding colonoscopy-specific augmentations to the training data can improve state-of-the-art polyp detection/segmentation methods.
  arXiv  Detail & Related papers  (2022-06-29T23:51:16Z)
• Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging [142.11622043078867]
  We propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST) for HSI reconstruction.
  arXiv  Detail & Related papers  (2022-05-20T11:37:44Z)
• Stepwise Feature Fusion: Local Guides Global [14.394421688712052]
  We propose a new State-Of-The-Art model for medical image segmentation, the SSFormer, which uses a pyramid Transformer encoder to improve the generalization ability of models. Our proposed Progressive Locality Decoder can be adapted to the pyramid Transformer backbone to emphasize local features and attention dispersion.
  arXiv  Detail & Related papers  (2022-03-07T10:36:38Z)
• Self-Supervised U-Net for Segmenting Flat and Sessile Polyps [63.62764375279861]
  Development of colorectal polyps is one of the earliest signs of cancer. Early detection and resection of polyps can greatly increase survival rate to 90%. Computer-Aided Diagnosis systems(CADx) has been proposed that detect polyps by processing the colonoscopic videos.
  arXiv  Detail & Related papers  (2021-10-17T09:31:20Z)
• CyTran: A Cycle-Consistent Transformer with Multi-Level Consistency for Non-Contrast to Contrast CT Translation [56.622832383316215]
  We propose a novel approach to translate unpaired contrast computed tomography (CT) scans to non-contrast CT scans. Our approach is based on cycle-consistent generative adversarial convolutional transformers, for short, CyTran. Our empirical results show that CyTran outperforms all competing methods.
  arXiv  Detail & Related papers  (2021-10-12T23:25:03Z)
• Automatic Polyp Segmentation via Multi-scale Subtraction Network [100.94922587360871]
  In clinical practice, precise polyp segmentation provides important information in the early detection of colorectal cancer. Most existing methods are based on U-shape structure and use element-wise addition or concatenation to fuse different level features progressively in decoder. We propose a multi-scale subtraction network (MSNet) to segment polyp from colonoscopy image.
  arXiv  Detail & Related papers  (2021-08-11T07:54:07Z)
• Visualizing Missing Surfaces In Colonoscopy Videos using Shared Latent Space Representations [6.187780920448871]
  Optical colonoscopy (OC) has a high miss rate due to a number of factors. We present a framework to visualize the missed regions per-frame during the colonoscopy.
  arXiv  Detail & Related papers  (2021-01-18T19:00:51Z)
• Colorectal Polyp Segmentation by U-Net with Dilation Convolution [9.840695333927496]
  Colorectal cancer (CRC) is one of the most commonly diagnosed cancers and a leading cause of cancer deaths in the United States. Currently, the most common way for colorectal polyp detection and precancerous pathology is the colonoscopy. We propose a novel end-to-end deep learning framework for the colorectal polyp segmentation.
  arXiv  Detail & Related papers  (2019-12-26T23:27:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.