Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions

• URL: http://arxiv.org/abs/2606.26634v1
• Date: Thu, 25 Jun 2026 05:55:26 GMT
• Title: Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions
• Abstract summary: Flow-guided temporal for Robust Operating Scenes (FAROS)<n>A flow-guided label framework that combines zero-shot segmentation-based mask propagation with optical flow estimation to overcome the limitations of appearance-based propagation under challenging surgical conditions.<n>A unified Transformer-based multi-task framework that jointly learns surgical phase recognition, step recognition, anticipation, instrument segmentation, and action recognition.
• Score: 4.361526134899725
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effective multi-task learning for surgical scene understanding is fundamentally hindered by annotation granularity mismatch; temporal workflow tasks such as phase recognition, step recognition and anticipation benefit from dense frame-level supervision, whereas pixel-level spatial tasks including instrument segmentation and action recognition are only sparsely annotated on selected keyframes due to prohibitive labeling costs. This supervision imbalance undermines shared representation learning and limits joint optimization across heterogeneous surgical tasks. To address this, we propose Flow-guided Annotation for Robust Operating Scenes (FAROS), a flow-guided label interpolation framework, that combines zero-shot segmentation-based mask propagation with optical flow estimation to overcome the limitations of appearance-based propagation under challenging surgical conditions such as occlusion, smoke, and motion blur, generating temporally consistent dense pseudo labels from sparse keyframe annotations. The densified instrument masks and action labels are integrated into a unified Transformer-based multi-task framework that jointly learns surgical phase recognition, step recognition, anticipation, instrument segmentation, and action recognition, enabling balanced optimization between dense temporal supervision and sparse spatial supervision. The label interpolation quality of FAROS is first validated on the DAVIS 2017 benchmark under a sparse ground-truth protocol, confirming robust propagation beyond the surgical domain. Extensive experiments on GraSP, MISAW, and AutoLaparo benchmarks further demonstrate that FAROS significantly improves cross-task representation learning and enhances holistic surgical scene understanding performance across spatio-temporal tasks.

Related papers

• From Spatial Semantics to Temporal Context: Leveraging Gaze Trajectory for Weakly Supervised Medical Image Segmentation [5.809537365593592]
  Eye tracking offers a cost-effective solution that can be naturally integrated into clinical.<n>We propose Trajectory-guided Uncertainty-aware Network (TrailNet)<n>TrailNet exploits gaze-supervised medical image segmentation from spatial semantics to temporal context by jointly leveraging fixations and trajectories.
  arXiv  Detail & Related papers  (2026-07-29T07:13:31Z)
• Towards Unified Surgical Scene Understanding:Bridging Reasoning and Grounding via MLLMs [32.18549469001096]
  SurgMLLM is a unified surgical scene understanding framework.<n>It bridges high-level reasoning and low-level visual grounding within a single model.<n>Extensive experiments show that SurgMLLM significantly advances surgical scene understanding.
  arXiv  Detail & Related papers  (2026-05-13T13:42:23Z)
• CurConMix+: A Unified Spatio-Temporal Framework for Hierarchical Surgical Workflow Understanding [1.0692208281858493]
  Surgical action triplet recognition aims to understand fine surgical behaviors by modeling interactions among instruments, actions, and anatomical targets.<n>Despite its clinical importance for workflow analysis and skill assessment, progress has been hindered by severe class imbalance, subtle visual variations, and the interdependence among triplet components.<n>This study builds upon CurConMix, a spatial representation framework, to address these challenges.
  arXiv  Detail & Related papers  (2026-01-18T08:42:10Z)
• Surgical Video Understanding with Label Interpolation [3.880707330499936]
  Robot-assisted surgery (RAS) has become a critical paradigm in modern surgery, promoting patient recovery and reducing the burden on surgeons.<n>Previous studies have predominantly focused on single-task approaches, but real surgical scenes involve complex temporal dynamics and diverse instrument interactions.<n>We propose a novel framework that combines optical flow-based segmentation label with multi-task learning.
  arXiv  Detail & Related papers  (2025-09-23T08:49:07Z)
• Context-aware TFL: A Universal Context-aware Contrastive Learning Framework for Temporal Forgery Localization [60.73623588349311]
  We propose a universal context-aware contrastive learning framework (UniCaCLF) for temporal forgery localization.<n>Our approach leverages supervised contrastive learning to discover and identify forged instants by means of anomaly detection.<n>An efficient context-aware contrastive coding is introduced to further push the limit of instant feature distinguishability between genuine and forged instants.
  arXiv  Detail & Related papers  (2025-06-10T06:40:43Z)
• FinePseudo: Improving Pseudo-Labelling through Temporal-Alignablity for Semi-Supervised Fine-Grained Action Recognition [57.17966905865054]
  Real-life applications of action recognition often require a fine-grained understanding of subtle movements. Existing semi-supervised action recognition has mainly focused on coarse-grained action recognition. We propose an Alignability-Verification-based Metric learning technique to effectively discriminate between fine-grained action pairs.
  arXiv  Detail & Related papers  (2024-09-02T20:08:06Z)
• Auxiliary Tasks Enhanced Dual-affinity Learning for Weakly Supervised Semantic Segmentation [79.05949524349005]
  We propose AuxSegNet+, a weakly supervised auxiliary learning framework to explore the rich information from saliency maps. We also propose a cross-task affinity learning mechanism to learn pixel-level affinities from the saliency and segmentation feature maps.
  arXiv  Detail & Related papers  (2024-03-02T10:03:21Z)
• Spatiotemporal Multi-scale Bilateral Motion Network for Gait Recognition [3.1240043488226967]
  In this paper, motivated by optical flow, the bilateral motion-oriented features are proposed. We develop a set of multi-scale temporal representations that force the motion context to be richly described at various levels of temporal resolution.
  arXiv  Detail & Related papers  (2022-09-26T01:36:22Z)
• Real-time landmark detection for precise endoscopic submucosal dissection via shape-aware relation network [51.44506007844284]
  We propose a shape-aware relation network for accurate and real-time landmark detection in endoscopic submucosal dissection surgery. We first devise an algorithm to automatically generate relation keypoint heatmaps, which intuitively represent the prior knowledge of spatial relations among landmarks. We then develop two complementary regularization schemes to progressively incorporate the prior knowledge into the training process.
  arXiv  Detail & Related papers  (2021-11-08T07:57:30Z)
• AASeg: Attention Aware Network for Real Time Semantic Segmentation [0.0]
  We propose AASeg, a novel Attention-Aware Network for real-time semantic segmentation.<n>We show that AASeg achieves a compelling trade-off between accuracy and efficiency, outperforming prior real-time methods.
  arXiv  Detail & Related papers  (2021-07-27T20:01:55Z)
• Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentation [88.49669148290306]
  We propose a novel weakly supervised multi-task framework called AuxSegNet to leverage saliency detection and multi-label image classification as auxiliary tasks. Inspired by their similar structured semantics, we also propose to learn a cross-task global pixel-level affinity map from the saliency and segmentation representations. The learned cross-task affinity can be used to refine saliency predictions and propagate CAM maps to provide improved pseudo labels for both tasks.
  arXiv  Detail & Related papers  (2021-07-25T11:39:58Z)
• Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer [57.18185972461453]
  We introduce for the first time in surgical workflow analysis Transformer to reconsider the ignored complementary effects of spatial and temporal features for accurate phase recognition. Our framework is lightweight and processes the hybrid embeddings in parallel to achieve a high inference speed.
  arXiv  Detail & Related papers  (2021-03-17T15:12:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.