Optimize Surgical Triplet Recognition: A Knowledge-Driven Mixture-of-Experts Solution

• URL: http://arxiv.org/abs/2608.22972v1
• Date: Mon, 24 Aug 2026 08:32:53 GMT
• Title: Optimize Surgical Triplet Recognition: A Knowledge-Driven Mixture-of-Experts Solution
• Abstract summary: We propose a Mixture-of-Experts-guided Co-Optimization framework powered by knowledge-driven learning.<n>We develop a coordinated gradient learning strategy to handle category-level conflicts.<n> Notably, inspired by surgical domain expertise, we introduce a knowledge-driven mixture-of-experts mechanism.
• Score: 52.149337961205624
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Surgical action triplet recognition constitutes a critical task in context-aware robot-assisted surgery, facilitating automatic surgical action perception by identifying instrument, verb, target, and their association. However, existing works struggle to analyze such complex surgical scenes due to three main issues: (1) component-level optimization conflicts caused by entangled feature spaces, (2) category-level optimization conflicts arising from severe data imbalance, and (3) lack of domain knowledge guidance that limits model interpretability and robustness. To address these challenges, we propose a Mixture-of-Experts-guided Co-Optimization (\textit{MoeCo}) framework powered by knowledge-driven learning. Within the co-optimization pipeline, to first mitigate component-level conflicts, we introduce a component-tailored adapter that disentangles task-specific features across spatial-temporal regimes, facilitating effective component specialization. Next, we develop a coordinated gradient learning strategy to handle category-level conflicts, which adaptively rebalances positive-negative gradients to enhance the perception of rare categories. Notably, inspired by surgical domain expertise, we introduce a knowledge-driven mixture-of-experts mechanism that dynamically integrates multimodal large language model-guided knowledge via activated experts, thereby enriching the co-optimization pipeline with more expressive and robust representations. Extensive experiments on the public CholecT45 and CholecT50 datasets confirm the effectiveness of the proposed co-optimization pipeline and the superiority of dynamic priors integration via the knowledge-driven mixture-of-experts mechanism.

Related papers

• Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild [49.77540427384148]
  We present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild.<n>To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms.<n>AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings.
  arXiv  Detail & Related papers  (2026-07-15T14:30:20Z)
• AMATA: Adaptive Multi-Agent Trajectory Alignment for Knowledge-Intensive Question Answering [59.145044878490815]
  AMATA is an Adaptive Multi-Agent Trajectory Alignment framework.<n>It dynamically integrates external knowledge to improve response interpretability and factual grounding.<n>We introduce two principal innovations: (1) Intra-Trajectory Preference Learning, which learns objective-oriented preferences to prioritize critical agents, and (2) Inter-Agent Dependency Learning, which captures cross-agent tool dependencies.
  arXiv  Detail & Related papers  (2026-05-17T09:45:24Z)
• CurConMix+: A Unified Spatio-Temporal Framework for Hierarchical Surgical Workflow Understanding [1.0692208281858493]
  Surgical action triplet recognition aims to understand fine surgical behaviors by modeling interactions among instruments, actions, and anatomical targets.<n>Despite its clinical importance for workflow analysis and skill assessment, progress has been hindered by severe class imbalance, subtle visual variations, and the interdependence among triplet components.<n>This study builds upon CurConMix, a spatial representation framework, to address these challenges.
  arXiv  Detail & Related papers  (2026-01-18T08:42:10Z)
• Heterogeneous Multi-Agent Reinforcement Learning with Attention for Cooperative and Scalable Feature Transformation [21.732611237889326]
  Feature transformation enhances downstream task performance by generating informative features through mathematical feature crossing.<n>Recent works employ reinforcement learning to enhance traditional approaches through a more effective trial-and-error way.<n>We propose a novel heterogeneous multi-agent RL framework to enable cooperative and scalable feature transformation.
  arXiv  Detail & Related papers  (2025-11-26T21:45:38Z)
• MEJO: MLLM-Engaged Surgical Triplet Recognition via Inter- and Intra-Task Joint Optimization [52.149337961205624]
  We propose a framework that empowers both inter- and intra-task optimization for surgical triplet recognition.<n>For inter-task optimization, we introduce the Shared-Specific-Disentangled (S$2$D) learning scheme that decomposes representations into task-shared and task-specific components.<n>For intra-task optimization conflicts, we develop a Coordinated Gradient Learning (CGL) strategy, which dissects and rebalances the positive-negative ambiguities.
  arXiv  Detail & Related papers  (2025-09-16T09:48:52Z)
• SurgRAW: Multi-Agent Workflow with Chain-of-Thought Reasoning for Surgical Intelligence [16.584722724845182]
  Integration of Vision-Language Models in surgical intelligence is hindered by hallucinations, domain knowledge gaps, and limited understanding of task interdependencies.<n>We present SurgRAW, a CoT-driven multi-agent framework that delivers transparent, interpretable insights for most tasks in robotic-assisted surgery.
  arXiv  Detail & Related papers  (2025-03-13T11:23:13Z)
• Astrea: A MOE-based Visual Understanding Model with Progressive Alignment [10.943104653307294]
  Vision-Language Models (VLMs) based on Mixture-of-Experts (MoE) architectures have emerged as a pivotal paradigm in multimodal understanding.<n>We propose Astrea, a novel multi-expert collaborative VLM architecture based on progressive pre-alignment.
  arXiv  Detail & Related papers  (2025-03-12T14:44:52Z)
• EAGER-LLM: Enhancing Large Language Models as Recommenders through Exogenous Behavior-Semantic Integration [60.47645731801866]
  Large language models (LLMs) are increasingly leveraged as foundational backbones in advanced recommender systems.<n>LLMs are pre-trained linguistic semantics but learn collaborative semantics from scratch via the llm-Backbone.<n>We propose EAGER-LLM, a decoder-only generative recommendation framework that integrates endogenous and endogenous behavioral and semantic information in a non-intrusive manner.
  arXiv  Detail & Related papers  (2025-02-20T17:01:57Z)
• Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection [57.13665112065285]
  Human-Object Interaction (HOI) detection is a challenging computer vision task. We present a framework that enhances HOI detection by incorporating structured text knowledge.
  arXiv  Detail & Related papers  (2023-07-25T14:20:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.