From Real Artifacts to Virtual Reference: A Robust Framework for Translating Endoscopic Images

• URL: http://arxiv.org/abs/2410.13896v2
• Date: Wed, 23 Oct 2024 13:01:22 GMT
• Title: From Real Artifacts to Virtual Reference: A Robust Framework for Translating Endoscopic Images
• Authors: Junyang Wu, Fangfang Xie, Jiayuan Sun, Yun Gu, Guang-Zhong Yang,
• Abstract summary: In endoscopic imaging, combining pre-operative data with intra-operative imaging is important for surgical planning and navigation. Existing domain adaptation methods are hampered by distribution shift caused by in vivo artifacts. This paper presents an artifact-resilient image translation method and an associated benchmark for this purpose.
• Score: 27.230439605570812
• License:
• Abstract: Domain adaptation, which bridges the distributions across different modalities, plays a crucial role in multimodal medical image analysis. In endoscopic imaging, combining pre-operative data with intra-operative imaging is important for surgical planning and navigation. However, existing domain adaptation methods are hampered by distribution shift caused by in vivo artifacts, necessitating robust techniques for aligning noisy and artifact abundant patient endoscopic videos with clean virtual images reconstructed from pre-operative tomographic data for pose estimation during intraoperative guidance. This paper presents an artifact-resilient image translation method and an associated benchmark for this purpose. The method incorporates a novel ``local-global'' translation framework and a noise-resilient feature extraction strategy. For the former, it decouples the image translation process into a local step for feature denoising, and a global step for global style transfer. For feature extraction, a new contrastive learning strategy is proposed, which can extract noise-resilient features for establishing robust correspondence across domains. Detailed validation on both public and in-house clinical datasets has been conducted, demonstrating significantly improved performance compared to the current state-of-the-art.

Related papers

• Intraoperative Registration by Cross-Modal Inverse Neural Rendering [61.687068931599846]
  We present a novel approach for 3D/2D intraoperative registration during neurosurgery via cross-modal inverse neural rendering. Our approach separates implicit neural representation into two components, handling anatomical structure preoperatively and appearance intraoperatively. We tested our method on retrospective patients' data from clinical cases, showing that our method outperforms state-of-the-art while meeting current clinical standards for registration.
  arXiv  Detail & Related papers  (2024-09-18T13:40:59Z)
• Denoising as Adaptation: Noise-Space Domain Adaptation for Image Restoration [64.84134880709625]
  We show that it is possible to perform domain adaptation via the noise space using diffusion models. In particular, by leveraging the unique property of how auxiliary conditional inputs influence the multi-step denoising process, we derive a meaningful diffusion loss. We present crucial strategies such as channel-shuffling layer and residual-swapping contrastive learning in the diffusion model.
  arXiv  Detail & Related papers  (2024-06-26T17:40:30Z)
• CathFlow: Self-Supervised Segmentation of Catheters in Interventional Ultrasound Using Optical Flow and Transformers [66.15847237150909]
  We introduce a self-supervised deep learning architecture to segment catheters in longitudinal ultrasound images. The network architecture builds upon AiAReSeg, a segmentation transformer built with the Attention in Attention mechanism. We validated our model on a test dataset, consisting of unseen synthetic data and images collected from silicon aorta phantoms.
  arXiv  Detail & Related papers  (2024-03-21T15:13:36Z)
• A2V: A Semi-Supervised Domain Adaptation Framework for Brain Vessel Segmentation via Two-Phase Training Angiography-to-Venography Translation [4.452428104996953]
  We present a semi-supervised domain adaptation framework for brain vessel segmentation from different image modalities. By relying on annotated angiographies and a limited number of annotated venographies, our framework accomplishes image-to-image translation and semantic segmentation.
  arXiv  Detail & Related papers  (2023-09-12T09:12:37Z)
• Domain Adaptive Sim-to-Real Segmentation of Oropharyngeal Organs [14.723143613743211]
  Video-assisted transoral tracheal intubation (TI) necessitates using an endoscope that helps the physician insert a tracheal tube into the glottis instead of the esophagus. The real of oropharyngeal organs are often inaccessible due to limited open-source data and patient privacy. We propose a domain adaptive Sim-to-Real framework called IoU-Ranking Blend-ArtFlow (IRB-AF) for image segmentation of oropharyngeal organs.
  arXiv  Detail & Related papers  (2023-05-18T11:25:23Z)
• Unsupervised Domain Transfer with Conditional Invertible Neural Networks [83.90291882730925]
  We propose a domain transfer approach based on conditional invertible neural networks (cINNs) Our method inherently guarantees cycle consistency through its invertible architecture, and network training can efficiently be conducted with maximum likelihood. Our method enables the generation of realistic spectral data and outperforms the state of the art on two downstream classification tasks.
  arXiv  Detail & Related papers  (2023-03-17T18:00:27Z)
• Orientation-Shared Convolution Representation for CT Metal Artifact Learning [63.67718355820655]
  During X-ray computed tomography (CT) scanning, metallic implants carrying with patients often lead to adverse artifacts. Existing deep-learning-based methods have gained promising reconstruction performance. We propose an orientation-shared convolution representation strategy to adapt the physical prior structures of artifacts.
  arXiv  Detail & Related papers  (2022-12-26T13:56:12Z)
• ScoreNet: Learning Non-Uniform Attention and Augmentation for Transformer-Based Histopathological Image Classification [11.680355561258427]
  High-resolution images hinder progress in digital pathology. patch-based processing often incorporates multiple instance learning (MIL) to aggregate local patch-level representations yielding image-level prediction. This paper proposes a transformer-based architecture specifically tailored for histological image classification. It combines fine-grained local attention with a coarse global attention mechanism to learn meaningful representations of high-resolution images at an efficient computational cost.
  arXiv  Detail & Related papers  (2022-02-15T16:55:09Z)
• Semantic segmentation of multispectral photoacoustic images using deep learning [53.65837038435433]
  Photoacoustic imaging has the potential to revolutionise healthcare. Clinical translation of the technology requires conversion of the high-dimensional acquired data into clinically relevant and interpretable information. We present a deep learning-based approach to semantic segmentation of multispectral photoacoustic images.
  arXiv  Detail & Related papers  (2021-05-20T09:33:55Z)
• Content-Preserving Unpaired Translation from Simulated to Realistic Ultrasound Images [12.136874314973689]
  We introduce a novel image translation framework to bridge the appearance gap between simulated images and real scans. We achieve this goal by leveraging both simulated images with semantic segmentations and unpaired in-vivo ultrasound scans.
  arXiv  Detail & Related papers  (2021-03-09T22:35:43Z)
• Self-Attentive Spatial Adaptive Normalization for Cross-Modality Domain Adaptation [9.659642285903418]
  Cross-modality synthesis of medical images to reduce the costly annotation burden by radiologists. We present a novel approach for image-to-image translation in medical images, capable of supervised or unsupervised (unpaired image data) setups.
  arXiv  Detail & Related papers  (2021-03-05T16:22:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.