OR-Action: Multi-Role Video Understanding with Fine-Grained Actions

• URL: http://arxiv.org/abs/2606.13332v1
• Date: Thu, 11 Jun 2026 13:24:40 GMT
• Title: OR-Action: Multi-Role Video Understanding with Fine-Grained Actions
• Abstract summary: Fine-grained understanding of operating room (OR) activity could enable workflow-aware assistance.<n>Current scene graph prediction methods struggle to model temporal structure.<n>We introduce a vision-only temporal model that outperforms graph-based methods.
• Score: 37.96856087875045
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-grained understanding of operating room (OR) activity could enable workflow-aware assistance, yet remains difficult due to clutter, occlusions, and limited sensing. The prevailing approach to model this environment is scene graphs as an interpretable representation of OR interactions. Converting their frame-wise relational predictions into temporally extended, fine-grained actions however, is challenging without explicit temporal modeling. To enable a principled temporal evaluation of current OR understanding methods, we introduce the first action-centric benchmark built on a publicly available ego-exocentric OR dataset by defining a fine-grained, multi-role action taxonomy and generating dense action segments via distillation from ground-truth scene graph state changes. Experiments on this benchmark show that current scene graph prediction methods struggle to model temporal structure, even when adding explicit modeling through Graph Neural Networks. We therefore introduce a vision-only temporal model that outperforms graph-based methods significantly when using all available egocentric video as input. Building on this model we also introduce a novel multi- to single-view feature alignment strategy that improves single-view performance on multi-role action recognition, mitigating the need for extensive egocentric video capture. Benchmark and code will be released upon acceptance.

Related papers

• Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs [10.367002095946843]
  We argue that explicit, compositional and editable interactions can play a crucial role for rich activity understanding.<n>We introduce EgoEgo, a large scale annotation set extending Ego4D with annotated scene graphs.<n>We also formulate the activity-driven graph-edit forecasting (A-GEF) problem, a novel task that casts scene dynamics as a sequence of structured transformations.
  arXiv  Detail & Related papers  (2026-07-02T16:53:42Z)
• InstrAct: Towards Action-Centric Understanding in Instructional Videos [12.356484522873577]
  InstrAction is a pretraining framework for instructional videos' action-centric representations.<n>We first introduce a data-driven strategy, which filters noisy captions and generates action-centric hard negatives.<n>At the visual feature level, an Action Perceiver extracts motion-relevant tokens from redundant video encodings.
  arXiv  Detail & Related papers  (2026-04-09T20:51:13Z)
• Exploring the Temporal Consistency for Point-Level Weakly-Supervised Temporal Action Localization [66.80402022104074]
  Point-supervised Temporal Action Localization (PTAL) adopts a lightly frame-annotated paradigm (textiti.e., labeling only a single frame per action instance) to train a model to locate action instances within unsupervised videos.<n>Most existing approaches design the task head of models with only a point-trimmed snippet-level classification, without explicit modeling of understanding temporal relationships among frames of an action.<n>We propose a multi-task learning framework that fully utilizes point supervision to boost the model's temporal understanding capability for action localization.
  arXiv  Detail & Related papers  (2026-02-05T14:46:21Z)
• THYME: Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graphs in Aerial Footage [11.587822611656648]
  We introduce the Temporal Hierarchical Cyclic Scene Graph (THYME) approach, which integrates hierarchical feature aggregation with cyclic temporal refinement to address limitations.<n>THYME effectively models multi-scale spatial context and enforces temporal consistency across frames, yielding more accurate and coherent scene graphs.<n>In addition, we present AeroEye-v1.0, a novel aerial video dataset enriched with five types of interactivity that overcomes the constraints of existing datasets.
  arXiv  Detail & Related papers  (2025-07-12T08:43:38Z)
• Leveraging Foundation Models for Multimodal Graph-Based Action Recognition [2.066890710233268]
  We introduce a graph-based framework that integrates a vision-temporal foundation leveraging VideoMAE for dynamic visual encoding and BERT for contextual textual embedding.<n>We show that our method consistently outperforms state-of-the-art baselines on diverse benchmark datasets.
  arXiv  Detail & Related papers  (2025-05-21T07:15:14Z)
• TimeGraphs: Graph-based Temporal Reasoning [64.18083371645956]
  TimeGraphs is a novel approach that characterizes dynamic interactions as a hierarchical temporal graph. Our approach models the interactions using a compact graph-based representation, enabling adaptive reasoning across diverse time scales. We evaluate TimeGraphs on multiple datasets with complex, dynamic agent interactions, including a football simulator, the Resistance game, and the MOMA human activity dataset.
  arXiv  Detail & Related papers  (2024-01-06T06:26:49Z)
• Local-Global Information Interaction Debiasing for Dynamic Scene Graph Generation [51.92419880088668]
  We propose a novel DynSGG model based on multi-task learning, DynSGG-MTL, which introduces the local interaction information and global human-action interaction information. Long-temporal human actions supervise the model to generate multiple scene graphs that conform to the global constraints and avoid the model being unable to learn the tail predicates.
  arXiv  Detail & Related papers  (2023-08-10T01:24:25Z)
• EasyDGL: Encode, Train and Interpret for Continuous-time Dynamic Graph Learning [92.71579608528907]
  This paper aims to design an easy-to-use pipeline (termed as EasyDGL) composed of three key modules with both strong ability fitting and interpretability. EasyDGL can effectively quantify the predictive power of frequency content that a model learn from the evolving graph data.
  arXiv  Detail & Related papers  (2023-03-22T06:35:08Z)
• Self-Regulated Learning for Egocentric Video Activity Anticipation [147.9783215348252]
  Self-Regulated Learning (SRL) aims to regulate the intermediate representation consecutively to produce representation that emphasizes the novel information in the frame of the current time-stamp. SRL sharply outperforms existing state-of-the-art in most cases on two egocentric video datasets and two third-person video datasets.
  arXiv  Detail & Related papers  (2021-11-23T03:29:18Z)
• With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition [95.99542238790038]
  We propose a method that learns to attend to surrounding actions in order to improve recognition performance. To incorporate the temporal context, we propose a transformer-based multimodal model that ingests video and audio as input modalities. We test our approach on EPIC-KITCHENS and EGTEA datasets reporting state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-11-01T15:27:35Z)
• Action Localization through Continual Predictive Learning [14.582013761620738]
  We present a new approach based on continual learning that uses feature-level predictions for self-supervision. We use a stack of LSTMs coupled with CNN encoder, along with novel attention mechanisms, to model the events in the video and use this model to predict high-level features for the future frames. This self-supervised framework is not complicated as other approaches but is very effective in learning robust visual representations for both labeling and localization.
  arXiv  Detail & Related papers  (2020-03-26T23:32:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.