Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins

• URL: http://arxiv.org/abs/2606.17298v1
• Date: Mon, 15 Jun 2026 21:06:53 GMT
• Title: Reasoning Text-to-Video Retrieval for Operating Room Clips via Action-Driven Digital Twins
• Abstract summary: We propose OR3, a text-to-video retrieval method that converts clips into action-driven digital twins (ActDTs)<n>We construct a benchmark from MM-OR with 276 implicit queries across four reasoning categories over 386 clips from robotic knee procedures.<n> OR3 achieves 57.6 R@1 and 77.3 R@5, outperforming the strongest baseline.
• Score: 10.60845088912767
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-video retrieval in operating rooms (OR) is an enabling technology for OR safety, as it allows stakeholders to retrieve and inspect recordings of specific events. However, because the most safety-critical events may not follow the common structure, to unlock its full potential text-to-video retrieval must be able to handle implicit queries that require reasoning to identify the right video (e.g., the step right before clipping). However, existing methods rely on global embeddings that cannot reason over such queries. We propose OR3, a text-to-video retrieval method that converts clips into action-driven digital twins (ActDTs), grouping concurrent subject-action-object triplets under non-overlapping temporal intervals. Moreover, rather than cross-modal matching through paired encoders, OR3 performs imagination-based retrieval where an LLM generates hypothetical ActDTs from queries. This enables intra-modal matching via a single encoder trained with ActDT-tailored hard negatives. Finally, evidence-grounded refinement revises imagined ActDTs based on discrepancies with top candidates to capture procedure-specific patterns. We construct a benchmark from MM-OR with 276 implicit queries across four reasoning categories over 386 clips from robotic knee procedures. OR3 achieves 57.6 R@1 and 77.3 R@5, outperforming the strongest baseline. These results demonstrate that OR3 enables fine-grained discrimination between visually similar OR video clips through temporal action reasoning.

Related papers

• VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement [31.30208652334818]
  VideoSearch-R1 is an agentic framework for iterative video retrieval and reasoning through multi-turn interaction with a video search engine.<n>VideoSearch-R1 achieves state-of-the-art performance across three datasets on Video Moment Retrieval (VCMR)
  arXiv  Detail & Related papers  (2026-07-01T04:59:24Z)
• MAVIS: Multi-Agent Video Retrieval via Structured Video Understanding [11.308633179211348]
  textbfMAVIS is a novel multi-agent framework that rethinks retrieval as cooperative reasoning rather than brute-force search.<n>During retrieval, a planner decomposes complex user intents into atomic sub-tasks, dispatching specialized agents to independently nominate candidates.<n>Experiments on MSR-VTT, MSVD, and ActivityNet demonstrate that MAVIS achieves competitive performance without task-specific fine-tuning.
  arXiv  Detail & Related papers  (2026-06-08T15:36:15Z)
• Driving Video Retrieval for Complex Queries with Structured Grounding [6.16173133327134]
  STRIVE-D is a data-calibrated retrieval framework for driving videos.<n>It uses weakly labeled in-domain videos to estimate when a query rule is reliable.<n>It delivers up to 84% relative improvement in top-1 accuracy over state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-06-08T07:00:33Z)
• R^3: Composed Video Retrieval via Reasoning-Guided Recalling and Re-ranking [73.0537447183962]
  We present a zero-shot composed video retrieval pipeline built around Reasoning-guided Recalling and Reranking.<n>The model generates a reasoning trace that describes the expected target video after applying the edit.<n>Experiments have demonstrated the effectiveness of our method in addressing this challenge.
  arXiv  Detail & Related papers  (2026-05-31T09:20:53Z)
• Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language [60.91064560080974]
  We propose an efficient approach, SpotVMR, to trim the query-relevant clip.<n>Our proposed SpotVMR can serve as plug-and-play module, which achieves efficiency for state-of-the-art VMR methods.
  arXiv  Detail & Related papers  (2026-05-28T11:42:28Z)
• Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval [70.47320120707029]
  Partially relevant video retrieval aims to retrieve untrimmed videos using text queries that describe only partial content.<n>In this setting, vague queries often induce semantic ambiguity across videos, a challenge that is further exacerbated by the sparse temporal supervision within videos.<n>We propose Holmes, a hierarchical evidential learning framework that aggregates multi-granular cross-modal evidence to quantify and model uncertainty explicitly.
  arXiv  Detail & Related papers  (2026-05-07T12:06:13Z)
• Video-BrowseComp: Benchmarking Agentic Video Research on Open Web [64.53060049124961]
  Video-BrowseComp is a benchmark comprising 210 questions tailored for open-web agentic video reasoning.<n>It enforces a mandatory dependency on temporal visual evidence, ensuring answers cannot be derived solely through text search.<n>As the first open-web video research benchmark, Video-BrowseComp advances the field beyond passive perception toward proactive video reasoning.
  arXiv  Detail & Related papers  (2025-12-28T19:08:27Z)
• Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Model [70.97446870672069]
  Video anomaly detection (VAD) has been paid increasing attention due to its potential applications. Video Anomaly Retrieval ( VAR) aims to pragmatically retrieve relevant anomalous videos by cross-modalities. We present two benchmarks, UCFCrime-AR and XD-Violence, constructed on top of prevalent anomaly datasets.
  arXiv  Detail & Related papers  (2023-07-24T06:22:37Z)
• Temporal Query Networks for Fine-grained Video Understanding [88.9877174286279]
  We cast this into a query-response mechanism, where each query addresses a particular question, and has its own response label set. We evaluate the method extensively on the FineGym and Diving48 benchmarks for fine-grained action classification and surpass the state-of-the-art using only RGB features.
  arXiv  Detail & Related papers  (2021-04-19T17:58:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.