Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train

• URL: http://arxiv.org/abs/2306.16741v4
• Date: Tue, 9 Jan 2024 11:30:41 GMT
• Title: Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train
• Abstract summary: We propose Endo-FM, a foundation model specifically developed using massive endoscopic video data. Our dataset overall consists of over 33K video clips with up to 5 million frames, encompassing various protocols, target organs, and disease types. Our Endo-FM surpasses the current state-of-the-art (SOTA) self-supervised pre-training and adapter-based transfer learning methods by a significant margin.
• Score: 27.188088885290025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Foundation models have exhibited remarkable success in various applications, such as disease diagnosis and text report generation. To date, a foundation model for endoscopic video analysis is still lacking. In this paper, we propose Endo-FM, a foundation model specifically developed using massive endoscopic video data. First, we build a video transformer, which captures both local and global long-range dependencies across spatial and temporal dimensions. Second, we pre-train our transformer model using global and local views via a self-supervised manner, aiming to make it robust to spatial-temporal variations and discriminative across different scenes. To develop the foundation model, we construct a large-scale endoscopy video dataset by combining 9 publicly available datasets and a privately collected dataset from Baoshan Branch of Renji Hospital in Shanghai, China. Our dataset overall consists of over 33K video clips with up to 5 million frames, encompassing various protocols, target organs, and disease types. Our pre-trained Endo-FM can be easily adopted for a given downstream task via fine-tuning by serving as the backbone. With experiments on 3 different types of downstream tasks, including classification, segmentation, and detection, our Endo-FM surpasses the current state-of-the-art (SOTA) self-supervised pre-training and adapter-based transfer learning methods by a significant margin, such as VCL (3.1% F1, 4.8% Dice, and 5.5% F1 for classification, segmentation, and detection) and ST-Adapter (5.9% F1, 9.6% Dice, and 9.9% F1 for classification, segmentation, and detection). Code, datasets, and models are released at https://github.com/med-air/Endo-FM.

Related papers

• SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models [88.64689505355055]
  We introduce SolarWM, a fully open foundation for building interactive video world models from data preparation through long-horizon inference.<n>SolarWM addresses this coupling with a reconfigurable multi-source data engine and a backbone-native adaptation framework.<n>We instantiate four 5B--33B models based on Wan2.2, LTX-2.5, and MiniMax-H3 while preserving their native representations and objectives.<n>The resulting causal models enable real-time interaction over rollouts ranging from minutes to hours after being trained on only 5s sequences.
  arXiv  Detail & Related papers  (2026-09-02T17:59:41Z)
• Adapting Foundation Models for Annotation-Efficient Adnexal Mass Segmentation in Cine Images [4.42778347374376]
  Adnexal mass evaluation via ultrasound is a challenging clinical task, often hindered by subjective interpretation and significant inter-observer variability.<n>We propose a label-efficient segmentation framework that leverages the robust semantic priors of a pretrained DINOv3 foundational vision transformer backbone.<n>Our method achieves state-of-the-art performance compared to established fully supervised baselines, including U-Net, U-Net++, DeepLabV3, and MAnet.
  arXiv  Detail & Related papers  (2026-04-09T09:48:50Z)
• Scaling Video Pretraining for Surgical Foundation Models [51.92777479821822]
  SurgRec is a scalable and reproducible pretraining recipe for surgical video understanding.<n>We curate a large corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery.
  arXiv  Detail & Related papers  (2026-03-31T16:31:25Z)
• Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis [0.7068165275780814]
  We use V-JEPA as the single-modality foundation of a multimodal model for minimally invasive surgery support.<n>We analyze how the model's downstream performance can benefit from finetuning on unlabeled surgical video data.<n>Our results thus demonstrate how surgical data science can leverage public, generic foundation models.
  arXiv  Detail & Related papers  (2025-09-08T16:04:19Z)
• Crucial-Diff: A Unified Diffusion Model for Crucial Image and Annotation Synthesis in Data-scarce Scenarios [65.97836905826145]
  scarcity of data in various scenarios, such as medical, industry and autonomous driving, leads to model overfitting and dataset imbalance.<n>We propose Crucial-Diff, a domain-agnostic framework designed to synthesize crucial samples.<n>Our framework generates diverse, high-quality training data, achieving a pixel-level AP of 83.63% and an F1-MAX of 78.12% on MVTec.
  arXiv  Detail & Related papers  (2025-07-14T04:41:38Z)
• vesselFM: A Foundation Model for Universal 3D Blood Vessel Segmentation [2.167228869154864]
  We present vesselFM, a foundation model designed specifically for the broad task of 3D blood vessel segmentation. Unlike previous models, vesselFM can effortlessly generalize to unseen domains.
  arXiv  Detail & Related papers  (2024-11-26T12:44:42Z)
• AI in Lung Health: Benchmarking Detection and Diagnostic Models Across Multiple CT Scan Datasets [0.33923727961771083]
  Lung cancer remains the leading cause of cancer-related mortality worldwide. With the growing integration of artificial intelligence into medical imaging, the development and evaluation of robust AI models require access to large, well-annotated datasets. We benchmark deep learning models for both 3D nodule detection and lung cancer classification.
  arXiv  Detail & Related papers  (2024-05-07T18:36:40Z)
• No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance [68.18779562801762]
  multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance. Our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found.
  arXiv  Detail & Related papers  (2024-04-04T17:58:02Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• Building Universal Foundation Models for Medical Image Analysis with Spatially Adaptive Networks [5.661631789478932]
  We propose a universal foundation model for medical image analysis that processes images with heterogeneous spatial properties using a unified structure. We pre-train a spatial adaptive visual tokenizer (SPAD-VT) and then a spatial adaptive Vision Transformer (SPAD-ViT) via masked image modeling (MIM) on 55 public medical image datasets. The experimental results on downstream medical image classification and segmentation tasks demonstrate the superior performance and label efficiency of our model.
  arXiv  Detail & Related papers  (2023-12-12T08:33:45Z)
• MIS-FM: 3D Medical Image Segmentation using Foundation Models Pretrained on a Large-Scale Unannotated Dataset [14.823114726604853]
  We propose a novel self-supervised learning strategy named Volume Fusion (VF) for pretraining 3D segmentation models. VF forces the model to predict the fusion coefficient of each voxel, which is formulated as a self-supervised segmentation task without manual annotations. experiments with different downstream segmentation targets including head and neck organs, thoracic/abdominal organs showed that our pretrained model largely outperformed training from scratch.
  arXiv  Detail & Related papers  (2023-06-29T13:22:13Z)
• Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
  We propose a learnable weight-based hybrid medical image segmentation approach. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-15T17:55:05Z)
• Universal Segmentation of 33 Anatomies [19.194539991903593]
  We present an approach for learning a single model that universally segments 33 anatomical structures. We learn such a model from a union of multiple datasets, with each dataset containing the images that are partially labeled. We evaluate our model on multiple open-source datasets, proving that our model has a good generalization performance.
  arXiv  Detail & Related papers  (2022-03-04T02:29:54Z)
• Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis [7.214195462426705]
  We introduce a novel self-supervised learning framework with tailored proxy tasks for medical image analysis. We demonstrate successful pre-training of the proposed model on 5,050 publicly available computed tomography (CT) images. Our model is currently the state-of-the-art (i.e. ranked 1st) on the public test leaderboards of both MSD and BTCV datasets.
  arXiv  Detail & Related papers  (2021-11-29T18:45:20Z)
• Learning Signal-Agnostic Manifolds of Neural Fields [50.066449953522685]
  We leverage neural fields to capture the underlying structure in image, shape, audio and cross-modal audiovisual domains. We show that by walking across the underlying manifold of GEM, we may generate new samples in our signal domains.
  arXiv  Detail & Related papers  (2021-11-11T18:57:40Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)
• Modeling Shared Responses in Neuroimaging Studies through MultiView ICA [94.31804763196116]
  Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization. We propose a novel MultiView Independent Component Analysis model for group studies, where data from each subject are modeled as a linear combination of shared independent sources plus noise. We demonstrate the usefulness of our approach first on fMRI data, where our model demonstrates improved sensitivity in identifying common sources among subjects.
  arXiv  Detail & Related papers  (2020-06-11T17:29:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.