SurgFed: Language-guided Multi-Task Federated Learning for Surgical Video Understanding

• URL: http://arxiv.org/abs/2603.09496v1
• Date: Tue, 10 Mar 2026 10:54:14 GMT
• Title: SurgFed: Language-guided Multi-Task Federated Learning for Surgical Video Understanding
• Abstract summary: SurgFed is a multi-task federated learning framework for surgical scene segmentation and depth estimation.<n>SurgFed is powered by two appealing designs, i.e., Language-guided Channel Selection and Language-guided Hyper Aggregation.<n>SurgFed yields improvements over the state-of-the-art methods in five public datasets across four surgical types.
• Score: 18.297136509444936
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Surgical scene Multi-Task Federated Learning (MTFL) is essential for robot-assisted minimally invasive surgery (RAS) but remains underexplored in surgical video understanding due to two key challenges: (1) Tissue Diversity: Local models struggle to adapt to site-specific tissue features, limiting their effectiveness in heterogeneous clinical environments and leading to poor local predictions. (2) Task Diversity: Server-side aggregation, relying solely on gradient-based clustering, often produces suboptimal or incorrect parameter updates due to inter-site task heterogeneity, resulting in inaccurate localization. In light of these two issues, we propose SurgFed, a multi-task federated learning framework, enabling federated learning for surgical scene segmentation and depth estimation across diverse surgical types. SurgFed is powered by two appealing designs, i.e., Language-guided Channel Selection (LCS) and Language-guided Hyper Aggregation (LHA), to address the challenge of fully exploration on corss-site and cross-task. Technically, the LCS is first designed a lightweight personalized channel selection network that enhances site-specific adaptation using pre-defined text inputs, which optimally the local model learn the specific embeddings. We further introduce the LHA that employs a layer-wise cross-attention mechanism with pre-defined text inputs to model task interactions across sites and guide a hypernetwork for personalized parameter updates. Extensive empirical evidence shows that SurgFed yields improvements over the state-of-the-art methods in five public datasets across four surgical types. The code is available at https://anonymous.4open.science/r/SurgFed-070E/.

Related papers

• HAAF: Hierarchical Adaptation and Alignment of Foundation Models for Few-Shot Pathology Anomaly Detection [10.649984141835189]
  We propose the Hierarchical Adaptation and Alignment Framework (HAAF)<n>At its core is a novel Cross-Level Scaled Alignment mechanism that enforces a sequential calibration order.<n>A dual-branch inference strategy integrates semantic scores with geometric prototypes to ensure stability in few-shot settings.
  arXiv  Detail & Related papers  (2026-01-24T10:31:21Z)
• Cataract-LMM: Large-Scale, Multi-Source, Multi-Task Benchmark for Deep Learning in Surgical Video Analysis [4.318540086708654]
  We present a dataset of 3,000 cataract surgery videos from two surgical centers, performed by surgeons with a range of experience levels.<n>This resource is enriched with four annotation layers: temporal surgical phases, instance segmentation of instruments and anatomical structures, instrument-tissue interaction tracking, and quantitative skill scores.<n>The technical quality of the dataset is supported by a series of benchmarking experiments for key surgical AI tasks.
  arXiv  Detail & Related papers  (2025-10-18T06:48:29Z)
• HiLa: Hierarchical Vision-Language Collaboration for Cancer Survival Prediction [55.00788339683146]
  We propose a novel Hierarchical vision-Language collaboration framework for improved survival prediction.<n> Specifically, HiLa employs pretrained feature extractors to generate hierarchical visual features from WSIs at both patch and region levels.<n>This ap-proach enables the comprehensive learning of discriminative visual features cor-responding to different survival-related attributes from prompts.
  arXiv  Detail & Related papers  (2025-07-07T02:06:25Z)
• SurgVidLM: Towards Multi-grained Surgical Video Understanding with Large Language Model [67.8359850515282]
  SurgVidLM is the first video language model designed to address both full and fine-grained surgical video comprehension.<n>We show that SurgVidLM significantly outperforms state-of-the-art Vid-LLMs of comparable parameter scale in both full and fine-grained video understanding tasks.
  arXiv  Detail & Related papers  (2025-06-22T02:16:18Z)
• Surgical Scene Segmentation by Transformer With Asymmetric Feature Enhancement [7.150163844454341]
  Vision-specific transformer method is a promising way for surgical scene understanding. We propose a novel Transformer-based framework with an Asymmetric Feature Enhancement module (TAFE) The proposed method outperforms the SOTA methods in several different surgical segmentation tasks and additionally proves its ability of fine-grained structure recognition.
  arXiv  Detail & Related papers  (2024-10-23T07:58:47Z)
• Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation [51.222684687924215]
  Surgical video-language pretraining faces unique challenges due to the knowledge domain gap and the scarcity of multi-modal data.<n>We propose a hierarchical knowledge augmentation approach and a novel Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining framework to tackle these issues.
  arXiv  Detail & Related papers  (2024-09-30T22:21:05Z)
• LLM-Assisted Multi-Teacher Continual Learning for Visual Question Answering in Robotic Surgery [57.358568111574314]
  Patient data privacy often restricts the availability of old data when updating the model. Prior CL studies overlooked two vital problems in the surgical domain. This paper proposes addressing these problems with a multimodal large language model (LLM) and an adaptive weight assignment methodology.
  arXiv  Detail & Related papers  (2024-02-26T15:35:24Z)
• Learning Multi-modal Representations by Watching Hundreds of Surgical Video Lectures [50.09187683845788]
  Recent advancements in surgical computer vision applications have been driven by vision-only models.<n>These methods rely on manually annotated surgical videos to predict a fixed set of object categories.<n>In this work, we put forward the idea that the surgical video lectures available through open surgical e-learning platforms can provide effective vision and language supervisory signals.
  arXiv  Detail & Related papers  (2023-07-27T22:38:12Z)
• Taxonomy Adaptive Cross-Domain Adaptation in Medical Imaging via Optimization Trajectory Distillation [73.83178465971552]
  The success of automated medical image analysis depends on large-scale and expert-annotated training sets. Unsupervised domain adaptation (UDA) has been raised as a promising approach to alleviate the burden of labeled data collection. We propose optimization trajectory distillation, a unified approach to address the two technical challenges from a new perspective.
  arXiv  Detail & Related papers  (2023-07-27T08:58:05Z)
• Identification of Cognitive Workload during Surgical Tasks with Multimodal Deep Learning [20.706268332427157]
  An increase in the associated Cognitive Workload (CWL) results from dealing with unexpected and repetitive tasks. In this paper, a cascade of two machine learning approaches is suggested for the multimodal recognition of CWL. A Convolutional Neural Network (CNN) uses this information to identify different types of CWL associated to each surgical task.
  arXiv  Detail & Related papers  (2022-09-12T18:29:34Z)
• BARNet: Bilinear Attention Network with Adaptive Receptive Fields for Surgical Instrument Segmentation [26.44585036105453]
  We propose a novel bilinear attention network with adaptive receptive field to solve these two challenges. The proposed network achieves the best performance 97.47% mean IOU on Cata7 and comes first place on EndoVis 2017 by 10.10% IOU overtaking second-ranking method.
  arXiv  Detail & Related papers  (2020-01-20T13:05:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.