Holistic Surgical Phase Recognition with Hierarchical Input Dependent State Space Models

• URL: http://arxiv.org/abs/2506.21330v1
• Date: Thu, 26 Jun 2025 14:43:57 GMT
• Title: Holistic Surgical Phase Recognition with Hierarchical Input Dependent State Space Models
• Abstract summary: We propose a novel hierarchical input-dependent state space model for surgical video analysis.<n>Our framework incorporates a temporally consistent visual feature extractor, which appends a state space model head to a visual feature extractor to propagate temporal information.<n> Experiments have shown that our method outperforms the current state-of-the-art methods by a large margin.
• Score: 56.2236083600999
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Surgical workflow analysis is essential in robot-assisted surgeries, yet the long duration of such procedures poses significant challenges for comprehensive video analysis. Recent approaches have predominantly relied on transformer models; however, their quadratic attention mechanism restricts efficient processing of lengthy surgical videos. In this paper, we propose a novel hierarchical input-dependent state space model that leverages the linear scaling property of state space models to enable decision making on full-length videos while capturing both local and global dynamics. Our framework incorporates a temporally consistent visual feature extractor, which appends a state space model head to a visual feature extractor to propagate temporal information. The proposed model consists of two key modules: a local-aggregation state space model block that effectively captures intricate local dynamics, and a global-relation state space model block that models temporal dependencies across the entire video. The model is trained using a hybrid discrete-continuous supervision strategy, where both signals of discrete phase labels and continuous phase progresses are propagated through the network. Experiments have shown that our method outperforms the current state-of-the-art methods by a large margin (+2.8% on Cholec80, +4.3% on MICCAI2016, and +12.9% on Heichole datasets). Code will be publicly available after paper acceptance.

Related papers

• Spatiotemporally Decoupled Autoregressive Diffusion Model for Human Motion Generation [19.887078319028102]
  We propose a unifiedtemporal decoupled framework named DeMoDiff to redesign motion representation and architecture.<n>Our framework achieves state-of-the-art reconstruction performance and compelling motion generation results.
  arXiv  Detail & Related papers  (2026-08-24T14:04:35Z)
• SphereVideo: Prototype-anchored Hyperspherical Boundary for Continual AI-generated Video Detection [61.05420454815012]
  AI-generated video (AIGV) detection aims to distinguish real videos from AI-generated ones.<n>In practice, detectors trained on existing data often fail to generalize to newly emerging generative models.<n>We propose SphereVideo, a novel framework for AIGV detection built on two key observations.
  arXiv  Detail & Related papers  (2026-08-02T15:54:54Z)
• Infinite Gaze Generation for Videos with Autoregressive Diffusion [37.82819999198602]
  We present a generative framework for infinite-horizon raw gaze prediction in videos of arbitrary length.<n>By leveraging an autoregressive diffusion model, we synthesize gaze trajectories characterized by continuous spatial coordinates and high-resolution timestamps.
  arXiv  Detail & Related papers  (2026-03-26T02:02:08Z)
• Context-Aware Multimodal Representation Learning for Spatio-Temporally Explicit Environmental Modelling [3.3984815208531014]
  We propose a representation learning framework that integrates different modalities into unified space at high-temporal resolution.<n>Our approach produces a latent space at native 10 m resolution and the temporal frequency of cloud-free Sentinel-2 data.<n>This enables the model to capture complementary remote sensing data and to preserve coherence across space and time.
  arXiv  Detail & Related papers  (2025-11-12T15:34:17Z)
• Towards Robust and Generalizable Continuous Space-Time Video Super-Resolution with Events [71.2439653098351]
  Continuous space-time video super-STVSR has garnered increasing interest for its capability to reconstruct high-resolution and high-frame-rate videos at arbitrary temporal scales.<n>We present EvEnhancer, a novel approach that marries unique properties of high temporal and high dynamic range encapsulated in event streams.<n>Our method achieves state-of-the-art performance on both synthetic and real-world datasets, while maintaining generalizability at OOD scales.
  arXiv  Detail & Related papers  (2025-10-04T15:23:07Z)
• Taming generative video models for zero-shot optical flow extraction [28.176290134216995]
  Self-supervised video models trained only for future frame prediction can be prompted, without fine-tuning, to output flow.<n>Inspired by the Counterfactual World Model (CWM) paradigm, we extend this idea to generative video models.<n> KL-tracing is a novel test-time procedure that injects a localized perturbation into the first frame, rolls out the model one step, and computes the Kullback-Leibler divergence between perturbed and ungenerative predictive distributions.
  arXiv  Detail & Related papers  (2025-07-11T23:59:38Z)
• VideoMolmo: Spatio-Temporal Grounding Meets Pointing [66.19964563104385]
  VideoMolmo is a model tailored for fine-grained pointing of video sequences.<n>A novel temporal mask fusion employs SAM2 for bidirectional point propagation.<n>To evaluate the generalization of VideoMolmo, we introduce VPoMolS-temporal, a challenging out-of-distribution benchmark spanning five real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-05T17:59:29Z)
• StateSpaceDiffuser: Bringing Long Context to Diffusion World Models [53.05314852577144]
  We introduce StateSpaceDiffuser, where a diffusion model is enabled to perform long-context tasks by integrating features from a state-space model.<n>This design restores long-term memory while preserving the high-fidelity synthesis of diffusion models.<n>Experiments show that StateSpaceDiffuser significantly outperforms a strong diffusion-only baseline.
  arXiv  Detail & Related papers  (2025-05-28T11:27:54Z)
• Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models [1.5678321653327674]
  We propose a two-stage, text-based method to generate high-fidelity surgical videos for under-represented classes.<n>We evaluate our method on two downstream tasks--action recognition and intra-operative event prediction-demonstrating.
  arXiv  Detail & Related papers  (2025-05-14T23:43:29Z)
• FreSca: Scaling in Frequency Space Enhances Diffusion Models [55.75504192166779]
  This paper explores frequency-based control within latent diffusion models.<n>We introduce FreSca, a novel framework that decomposes noise difference into low- and high-frequency components.<n>FreSca operates without any model retraining or architectural change, offering model- and task-agnostic control.
  arXiv  Detail & Related papers  (2025-04-02T22:03:11Z)
• SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answering [5.016335384639901]
  Multi-modal input of Audio-Visual Question Answering (AVQA) makes feature extraction and fusion processes more challenging. We propose SHMamba: Structured Hyperbolic State Space Model to integrate the advantages of hyperbolic geometry and state space models. Our method demonstrates superiority among all current major methods and is more suitable for practical application scenarios.
  arXiv  Detail & Related papers  (2024-06-14T08:43:31Z)
• Efficient Global-Local Memory for Real-time Instrument Segmentation of Robotic Surgical Video [53.14186293442669]
  We identify two important clues for surgical instrument perception, including local temporal dependency from adjacent frames and global semantic correlation in long-range duration. We propose a novel dual-memory network (DMNet) to relate both global and local-temporal knowledge. Our method largely outperforms the state-of-the-art works on segmentation accuracy while maintaining a real-time speed.
  arXiv  Detail & Related papers  (2021-09-28T10:10:14Z)
• Temporal Memory Relation Network for Workflow Recognition from Surgical Video [53.20825496640025]
  We propose a novel end-to-end temporal memory relation network (TMNet) for relating long-range and multi-scale temporal patterns. We have extensively validated our approach on two benchmark surgical video datasets.
  arXiv  Detail & Related papers  (2021-03-30T13:20:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.