Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis

• URL: http://arxiv.org/abs/2509.06831v1
• Date: Mon, 08 Sep 2025 16:04:19 GMT
• Title: Leveraging Generic Foundation Models for Multimodal Surgical Data Analysis
• Abstract summary: We use V-JEPA as the single-modality foundation of a multimodal model for minimally invasive surgery support.<n>We analyze how the model's downstream performance can benefit from finetuning on unlabeled surgical video data.<n>Our results thus demonstrate how surgical data science can leverage public, generic foundation models.
• Score: 0.7068165275780814
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate how both the adaptation of a generic foundation model via transfer learning and the integration of complementary modalities from the operating room (OR) can support surgical data science. To this end, we use V-JEPA as the single-modality foundation of a multimodal model for minimally invasive surgery support. We analyze how the model's downstream performance can benefit (a) from finetuning on unlabeled surgical video data and (b) from providing additional time-resolved data streams from the OR in a multimodal setup. In an in-house dataset of liver surgery videos, we analyze the tasks of predicting hospital length of stay and postoperative complications. In videos of the public HeiCo dataset, we analyze the task of surgical phase recognition. As a baseline, we apply pretrained V-JEPA to all tasks. We then finetune it on unlabeled, held-out videos to investigate its change in performance after domain adaptation. Following the idea of modular decision support networks, we integrate additional data streams from the OR by training a separate encoder to form a shared representation space with V-JEPA's embeddings. Our experiments show that finetuning on domain-specific data increases model performance. On the in-house data, integrating additional time-resolved data likewise benefits the model. On the HeiCo data, accuracy of the pretrained video-only, single-modality baseline setup is on par with the top-performing submissions of the EndoVis2017 challenge, while finetuning on domain-specific data increases accuracy further. Our results thus demonstrate how surgical data science can leverage public, generic foundation models. Likewise, they indicate the potential of domain adaptation and of integrating suitable complementary data streams from the OR. To support further research, we release our code and model weights at https://github.com/DigitalSurgeryLab-Basel/ML-CDS-2025.

Related papers

• GenMed: A Pairwise Generative Reformulation of Medical Diagnostic Tasks [32.49998442428115]
  We model the joint distribution $P(X,Y)$ using diffusion models and reframe inference as a test-time output optimization problem.<n>Our framework enables flexible, gradient-based conditioning at inference time without architectural changes or retraining.<n>Our results highlight the versatility of generative joint modeling as a foundation for reusable, task-agnostic medical AI systems.
  arXiv  Detail & Related papers  (2026-05-11T14:32:09Z)
• Scaling Video Pretraining for Surgical Foundation Models [51.92777479821822]
  SurgRec is a scalable and reproducible pretraining recipe for surgical video understanding.<n>We curate a large corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery.
  arXiv  Detail & Related papers  (2026-03-31T16:31:25Z)
• Towards Robust In-Context Learning for Medical Image Segmentation via Data Synthesis [12.827413884809644]
  In-Context Learning (ICL) for universal medical image segmentation has introduced an unprecedented demand for large-scale, diverse datasets for training.<n>We propose textbf SynthICL, a novel data synthesis framework built upon domain randomization.<n>Our work helps mitigate the data bottleneck for ICL-based segmentation, paving the way for robust models.
  arXiv  Detail & Related papers  (2025-09-24T02:44:53Z)
• Data Augmentation with Diffusion Models for Colon Polyp Localization on the Low Data Regime: How much real data is enough? [38.635356845350394]
  We conduct a set of experiments training different diffusion models that can generate jointly colonoscopy images with localization annotations.<n>The generated data is used on various transfer learning experiments in the task of polyp localization with a model based on YOLO v9 on the low data regime.
  arXiv  Detail & Related papers  (2024-11-28T05:25:33Z)
• Day-Night Adaptation: An Innovative Source-free Adaptation Framework for Medical Image Segmentation [51.520294290813865]
  We propose a novel adaptation framework called Day-Night Adaptation (DyNA) with insights.<n>During the day, a low-frequency prompt is trained to adapt the frozen model to each test sample.<n>During the night, we reuse test data collected from the day and introduce a global student model to bridge the knowledge between teacher and student models.
  arXiv  Detail & Related papers  (2024-10-17T12:02:29Z)
• ASPS: Augmented Segment Anything Model for Polyp Segmentation [77.25557224490075]
  The Segment Anything Model (SAM) has introduced unprecedented potential for polyp segmentation. SAM's Transformer-based structure prioritizes global and low-frequency information. CFA integrates a trainable CNN encoder branch with a frozen ViT encoder, enabling the integration of domain-specific knowledge.
  arXiv  Detail & Related papers  (2024-06-30T14:55:32Z)
• Source-Free Collaborative Domain Adaptation via Multi-Perspective Feature Enrichment for Functional MRI Analysis [55.03872260158717]
  Resting-state MRI functional (rs-fMRI) is increasingly employed in multi-site research to aid neurological disorder analysis. Many methods have been proposed to reduce fMRI heterogeneity between source and target domains. But acquiring source data is challenging due to concerns and/or data storage burdens in multi-site studies. We design a source-free collaborative domain adaptation framework for fMRI analysis, where only a pretrained source model and unlabeled target data are accessible.
  arXiv  Detail & Related papers  (2023-08-24T01:30:18Z)
• Foundation Model for Endoscopy Video Analysis via Large-scale Self-supervised Pre-train [27.188088885290025]
  We propose Endo-FM, a foundation model specifically developed using massive endoscopic video data. Our dataset overall consists of over 33K video clips with up to 5 million frames, encompassing various protocols, target organs, and disease types. Our Endo-FM surpasses the current state-of-the-art (SOTA) self-supervised pre-training and adapter-based transfer learning methods by a significant margin.
  arXiv  Detail & Related papers  (2023-06-29T07:34:25Z)
• Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
  We propose a learnable weight-based hybrid medical image segmentation approach. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-15T17:55:05Z)
• Convolutional Monge Mapping Normalization for learning on sleep data [63.22081662149488]
  We propose a new method called Convolutional Monge Mapping Normalization (CMMN) CMMN consists in filtering the signals in order to adapt their power spectrum density (PSD) to a Wasserstein barycenter estimated on training data. Numerical experiments on sleep EEG data show that CMMN leads to significant and consistent performance gains independent from the neural network architecture.
  arXiv  Detail & Related papers  (2023-05-30T08:24:01Z)
• Longitudinal Variational Autoencoder [1.4680035572775534]
  A common approach to analyse high-dimensional data that contains missing values is to learn a low-dimensional representation using variational autoencoders (VAEs) Standard VAEs assume that the learnt representations are i.i.d., and fail to capture the correlations between the data samples. We propose the Longitudinal VAE (L-VAE), that uses a multi-output additive Gaussian process (GP) prior to extend the VAE's capability to learn structured low-dimensional representations. Our approach can simultaneously accommodate both time-varying shared and random effects, produce structured low-dimensional representations
  arXiv  Detail & Related papers  (2020-06-17T10:30:14Z)
• Modeling Shared Responses in Neuroimaging Studies through MultiView ICA [94.31804763196116]
  Group studies involving large cohorts of subjects are important to draw general conclusions about brain functional organization. We propose a novel MultiView Independent Component Analysis model for group studies, where data from each subject are modeled as a linear combination of shared independent sources plus noise. We demonstrate the usefulness of our approach first on fMRI data, where our model demonstrates improved sensitivity in identifying common sources among subjects.
  arXiv  Detail & Related papers  (2020-06-11T17:29:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.