Sterilizable Scene Graph Generation for Operating Rooms

• URL: http://arxiv.org/abs/2608.16469v1
• Date: Mon, 17 Aug 2026 12:06:49 GMT
• Title: Sterilizable Scene Graph Generation for Operating Rooms
• Abstract summary: We introduce SG-NCA, a lightweight scene graph generation framework based on Neural Cellular Automata (NCA)<n>We evaluate SG-NCA on videos of cataract surgery and cholecystectomy, demonstrating performance comparable to established baselines while requiring 55x fewer parameters.
• Score: 1.46555267748668
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scene graph generation from surgical video enables a holistic and structured understanding of surgical scenes by modeling objects and their semantic relationships. Despite recent advances, state-of-the-art approaches rely on large, parameter-heavy deep learning models that are impractical for deployment in the operating room (OR) due to hardware footprint, hygiene constraints, latency, and data privacy concerns. To the best of our knowledge, this is the first scene graph generation method built on NCAs and the first NCA framework capable of learning structured representations. We introduce SG-NCA, a lightweight scene graph generation framework based on Neural Cellular Automata (NCA), designed for inference in fanless devices critical for OR hygiene protocols. SG-NCA is the first scene graph generation combining NCA-based multiclass segmentation for efficient object detection and feature extraction with a lightweight relation predictor. We evaluate SG-NCA on videos of cataract surgery and cholecystectomy, demonstrating performance comparable to established baselines while requiring 55x fewer parameters. We showcase deployment on fanless edge devices better suited for the OR and demonstrate downstream applications such as surgical video captioning, highlighting SG-NCA's potential for affordable, privacy-preserving, and OR-ready intraoperative scene understanding.

Related papers

• Surgical Anatomy Recognition with Context Learning using Foundation Representations [9.888703668362641]
  Accurate recognition of anatomical structures is essential for safe and effective minimally invasive surgery (MIS)<n>We present a combined dataset and model framework to advance anatomy-aware perception in MIS.
  arXiv  Detail & Related papers  (2026-06-20T16:05:28Z)
• Data-Efficient Surgical Phase Segmentation in Small-Incision Cataract Surgery: A Controlled Study of Vision Foundation Models [10.211838396947236]
  We study data-efficient phase segmentation for manual small-incision cataract surgery (SICS) through a controlled comparison of visual representations.<n>We compare supervised encoders (ResNet-50, I3D) against large self-supervised foundation models (DINOv3, V-JEPA2)<n>Foundation-model features improve segmentation performance in this setup, with DINOv3 ViT-7B achieving the best overall results.
  arXiv  Detail & Related papers  (2026-04-12T08:07:02Z)
• Scaling Video Pretraining for Surgical Foundation Models [51.92777479821822]
  SurgRec is a scalable and reproducible pretraining recipe for surgical video understanding.<n>We curate a large corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery.
  arXiv  Detail & Related papers  (2026-03-31T16:31:25Z)
• Surgical Scene Segmentation using a Spike-Driven Video Transformer with Real-Time Potential [26.958261975749974]
  We propose textitSpikeSurgSeg, the first spike-driven video Transformer framework tailored for surgical scene segmentation.<n>SpikeSurgSeg achieves most mIoU comparable to SOTA ANN-based models while reducing inference latency by at least $8times$.
  arXiv  Detail & Related papers  (2025-12-24T17:05:09Z)
• Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgery [36.192962258966105]
  Scene graphs (SGs) provide structured representations crucial for decoding complex, dynamic surgical environments.<n>This PRISMA-ScR-guided scoping review systematically maps the evolving landscape of SG research in surgery.<n>Our analysis reveals rapid growth, yet uncovers a critical 'data divide'<n>SGs are maturing into an essential semantic bridge, enabling a new generation of intelligent systems to improve surgical safety, efficiency, and training.
  arXiv  Detail & Related papers  (2025-09-25T09:25:46Z)
• EndoGen: Conditional Autoregressive Endoscopic Video Generation [51.97720772069513]
  We propose the first conditional endoscopic video generation framework, namely EndoGen.<n>Specifically, we build an autoregressive model with a tailored Spatiotemporal Grid-Frame Patterning strategy.<n>We demonstrate the effectiveness of our framework in generating high-quality, conditionally guided endoscopic content.
  arXiv  Detail & Related papers  (2025-07-23T10:32:20Z)
• Surgical Foundation Model Leveraging Compression and Entropy Maximization for Image-Guided Surgical Assistance [50.486523249499115]
  Real-time video understanding is critical to guide procedures in minimally invasive surgery (MIS)<n>We propose Compress-to-Explore (C2E), a novel self-supervised framework to learn compact, informative representations from surgical videos.<n>C2E uses entropy-maximizing decoders to compress images while preserving clinically relevant details, improving encoder performance without labeled data.
  arXiv  Detail & Related papers  (2025-05-16T14:02:24Z)
• SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow Prediction [37.86132786212667]
  We introduce an end-to-end framework for the generation and optimization of surgical scene graphs. Our solution outperforms the SOTA on the CATARACTS dataset by 8% accuracy and 10% F1 score in surgical workflow.
  arXiv  Detail & Related papers  (2024-07-29T17:44:34Z)
• Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs [79.64891686479213]
  We show that it is possible to improve vision and language models (VLMs) when learning from scene graphs (SGs) For the visual side, we incorporate a special "SG Component" in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions. Our method improves the performance of several popular VLMs on multiple datasets with only a mild degradation in ZS capabilities.
  arXiv  Detail & Related papers  (2023-05-10T17:52:26Z)
• LASER: A Neuro-Symbolic Framework for Learning Spatial-Temporal Scene Graphs with Weak Supervision [58.6039004982056]
  We propose a neuro-symbolic framework to enable training generators using only video captions.<n>An alignment algorithm overcomes the challenges of weak supervision by leveraging a differentiable symbolic reasoner.<n>We evaluate our method on three video datasets: OpenPVSG, 20BN, and MUGEN.
  arXiv  Detail & Related papers  (2023-04-15T22:24:05Z)
• Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation [116.87918100031153]
  We propose a Cross-modal clinical Graph Transformer (CGT) for ophthalmic report generation (ORG) CGT injects clinical relation triples into the visual features as prior knowledge to drive the decoding procedure. Experiments on the large-scale FFA-IR benchmark demonstrate that the proposed CGT is able to outperform previous benchmark methods.
  arXiv  Detail & Related papers  (2022-06-04T13:16:30Z)
• Multimodal Semantic Scene Graphs for Holistic Modeling of Surgical Procedures [70.69948035469467]
  We take advantage of the latest computer vision methodologies for generating 3D graphs from camera views. We then introduce the Multimodal Semantic Graph Scene (MSSG) which aims at providing unified symbolic and semantic representation of surgical procedures.
  arXiv  Detail & Related papers  (2021-06-09T14:35:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.