PolypSegTrack: Unified Foundation Model for Colonoscopy Video Analysis

• URL: http://arxiv.org/abs/2503.24108v2
• Date: Wed, 02 Apr 2025 19:58:56 GMT
• Title: PolypSegTrack: Unified Foundation Model for Colonoscopy Video Analysis
• Authors: Anwesa Choudhuri, Zhongpai Gao, Meng Zheng, Benjamin Planche, Terrence Chen, Ziyan Wu,
• Abstract summary: PolypSegTrack is a novel foundation model that jointly addresses polyp detection, segmentation, classification and unsupervised tracking in colonoscopic videos.<n>Our approach leverages a novel conditional mask loss, enabling flexible training across datasets with either pixel-level segmentation masks or bounding box annotations.<n>Our unsupervised tracking module reliably associates polyp instances across frames using object queries, without relying on any visions.
• Score: 28.764513004699676
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Early detection, accurate segmentation, classification and tracking of polyps during colonoscopy are critical for preventing colorectal cancer. Many existing deep-learning-based methods for analyzing colonoscopic videos either require task-specific fine-tuning, lack tracking capabilities, or rely on domain-specific pre-training. In this paper, we introduce PolypSegTrack, a novel foundation model that jointly addresses polyp detection, segmentation, classification and unsupervised tracking in colonoscopic videos. Our approach leverages a novel conditional mask loss, enabling flexible training across datasets with either pixel-level segmentation masks or bounding box annotations, allowing us to bypass task-specific fine-tuning. Our unsupervised tracking module reliably associates polyp instances across frames using object queries, without relying on any heuristics. We leverage a robust vision foundation model backbone that is pre-trained unsupervisedly on natural images, thereby removing the need for domain-specific pre-training. Extensive experiments on multiple polyp benchmarks demonstrate that our method significantly outperforms existing state-of-the-art approaches in detection, segmentation, classification, and tracking.

Related papers

• EndoFinder: Online Image Retrieval for Explainable Colorectal Polyp Diagnosis [10.83700068295662]
  EndoFinder is a content-based image retrieval framework. It finds the 'digital twin' polyp in the reference database given a newly detected polyp. The clinical semantics of the new polyp can be inferred referring to the matched ones.
  arXiv  Detail & Related papers  (2024-07-16T05:40:17Z)
• SSTFB: Leveraging self-supervised pretext learning and temporal self-attention with feature branching for real-time video polyp segmentation [4.027361638728112]
  We propose a video polyp segmentation method that performs self-supervised learning as an auxiliary task and a spatial-temporal self-attention mechanism for improved representation learning. Our experimental results demonstrate an improvement with respect to several state-of-the-art (SOTA) methods. Our ablation study confirms that the choice of the proposed joint end-to-end training improves network accuracy by over 3% and nearly 10% on both the Dice similarity coefficient and intersection-over-union.
  arXiv  Detail & Related papers  (2024-06-14T17:33:11Z)
• ECC-PolypDet: Enhanced CenterNet with Contrastive Learning for Automatic Polyp Detection [88.4359020192429]
  Existing methods either involve computationally expensive context aggregation or lack prior modeling of polyps, resulting in poor performance in challenging cases. In this paper, we propose the Enhanced CenterNet with Contrastive Learning (ECC-PolypDet), a two-stage training & end-to-end inference framework. Box-assisted Contrastive Learning (BCL) during training to minimize the intra-class difference and maximize the inter-class difference between foreground polyps and backgrounds, enabling our model to capture concealed polyps. In the fine-tuning stage, we introduce the IoU-guided Sample Re-weighting
  arXiv  Detail & Related papers  (2024-01-10T07:03:41Z)
• Self-Supervised Polyp Re-Identification in Colonoscopy [1.9678816712224196]
  We propose a robust long term polyp tracking method based on re-identification by visual appearance. Our solution uses an attention-based self-supervised ML model, specifically designed to leverage the temporal nature of video input.
  arXiv  Detail & Related papers  (2023-06-14T15:53:54Z)
• YONA: You Only Need One Adjacent Reference-frame for Accurate and Fast Video Polyp Detection [80.68520401539979]
  textbfYONA (textbfYou textbfOnly textbfNeed one textbfAdjacent Reference-frame) is an efficient end-to-end training framework for video polyp detection. Our proposed YONA outperforms previous state-of-the-art competitors by a large margin in both accuracy and speed.
  arXiv  Detail & Related papers  (2023-06-06T13:53:15Z)
• Unsupervised Learning of Accurate Siamese Tracking [68.58171095173056]
  We present a novel unsupervised tracking framework, in which we can learn temporal correspondence both on the classification branch and regression branch. Our tracker outperforms preceding unsupervised methods by a substantial margin, performing on par with supervised methods on large-scale datasets such as TrackingNet and LaSOT.
  arXiv  Detail & Related papers  (2022-04-04T13:39:43Z)
• Contrastive Transformer-based Multiple Instance Learning for Weakly Supervised Polyp Frame Detection [30.51410140271929]
  Current polyp detection methods from colonoscopy videos use exclusively normal (i.e., healthy) training images. We formulate polyp detection as a weakly-supervised anomaly detection task that uses video-level labelled training data to detect frame-level polyps.
  arXiv  Detail & Related papers  (2022-03-23T01:30:48Z)
• Self-Supervised Predictive Convolutional Attentive Block for Anomaly Detection [97.93062818228015]
  We propose to integrate the reconstruction-based functionality into a novel self-supervised predictive architectural building block. Our block is equipped with a loss that minimizes the reconstruction error with respect to the masked area in the receptive field. We demonstrate the generality of our block by integrating it into several state-of-the-art frameworks for anomaly detection on image and video.
  arXiv  Detail & Related papers  (2021-11-17T13:30:31Z)
• Automatic Polyp Segmentation via Multi-scale Subtraction Network [100.94922587360871]
  In clinical practice, precise polyp segmentation provides important information in the early detection of colorectal cancer. Most existing methods are based on U-shape structure and use element-wise addition or concatenation to fuse different level features progressively in decoder. We propose a multi-scale subtraction network (MSNet) to segment polyp from colonoscopy image.
  arXiv  Detail & Related papers  (2021-08-11T07:54:07Z)
• Colonoscopy Polyp Detection: Domain Adaptation From Medical Report Images to Real-time Videos [76.37907640271806]
  We propose an Image-video-joint polyp detection network (Ivy-Net) to address the domain gap between colonoscopy images from historical medical reports and real-time videos. Experiments on the collected dataset demonstrate that our Ivy-Net achieves the state-of-the-art result on colonoscopy video.
  arXiv  Detail & Related papers  (2020-12-31T10:33:09Z)
• Self-supervised Human Detection and Segmentation via Multi-view Consensus [116.92405645348185]
  We propose a multi-camera framework in which geometric constraints are embedded in the form of multi-view consistency during training. We show that our approach outperforms state-of-the-art self-supervised person detection and segmentation techniques on images that visually depart from those of standard benchmarks.
  arXiv  Detail & Related papers  (2020-12-09T15:47:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.