Data-centric Design of Learning-based Surgical Gaze Perception Models in Multi-Task Simulation

• URL: http://arxiv.org/abs/2602.09259v1
• Date: Mon, 09 Feb 2026 22:52:59 GMT
• Title: Data-centric Design of Learning-based Surgical Gaze Perception Models in Multi-Task Simulation
• Abstract summary: In robot-assisted minimally invasive surgery (RMIS), reduced haptic feedback and depth cues increase reliance on expert visual perception.<n>We introduce a paired active-passive, multi-task surgical gaze dataset collected on the da Vinci SimNow simulator across four drills.<n>We quantify skill- and modality-dependent differences in gaze organization and evaluate the substitutability of passive gaze for operative supervision.
• Score: 16.689550165317765
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In robot-assisted minimally invasive surgery (RMIS), reduced haptic feedback and depth cues increase reliance on expert visual perception, motivating gaze-guided training and learning-based surgical perception models. However, operative expert gaze is costly to collect, and it remains unclear how the source of gaze supervision, both expertise level (intermediate vs. novice) and perceptual modality (active execution vs. passive viewing), shapes what attention models learn. We introduce a paired active-passive, multi-task surgical gaze dataset collected on the da Vinci SimNow simulator across four drills. Active gaze was recorded during task execution using a VR headset with eye tracking, and the corresponding videos were reused as stimuli to collect passive gaze from observers, enabling controlled same-video comparisons. We quantify skill- and modality-dependent differences in gaze organization and evaluate the substitutability of passive gaze for operative supervision using fixation density overlap analyses and single-frame saliency modeling. Across settings, MSI-Net produced stable, interpretable predictions, whereas SalGAN was unstable and often poorly aligned with human fixations. Models trained on passive gaze recovered a substantial portion of intermediate active attention, but with predictable degradation, and transfer was asymmetric between active and passive targets. Notably, novice passive labels approximated intermediate-passive targets with limited loss on higher-quality demonstrations, suggesting a practical path for scalable, crowd-sourced gaze supervision in surgical coaching and perception modeling.

Related papers

• Towards Generalizable Visually Grounded Exploration of Household Devices [74.66117923766632]
  Key bottleneck of realizing general autonomous embodied agents lies in Generalizable Visually Grounded Exploration.<n>We introduce VGEBench, a benchmark designed to evaluate the generalizable visually grounded exploration capabilities of Vision-Language Models.<n>Unlike static datasets, we construct a Logic-Driven State Machine framework. This framework simulates multi-turn interaction loops, compelling agents to achieve goals by active visual perception and feedback-driven correction.
  arXiv  Detail & Related papers  (2026-09-01T07:42:45Z)
• EveLoad: Cognitive Workload Recognition from Event-Based Eye Movements [4.802081767877088]
  Existing eye-movement-based cognitive workload recognition methods rely on frame-based eye trackers.<n>Event cameras provide microsecond-level temporal resolution, high dynamic range and low latency, making them suitable for capturing fine-grained ocular dynamics.<n>We introduce EveLoad, which is the first event-based eye-movement dataset with graded cognitive workload annotations.
  arXiv  Detail & Related papers  (2026-06-23T21:06:40Z)
• Cooperative Informative Sensing for Monitoring Dynamic Indoor Environments via Multi-Agent Reinforcement Learning [56.64821510576244]
  We formulate cooperative active observation as a decentralized control problem in which multiple robots adjust their motion to directly optimize monitoring accuracy under partial observability.<n>We propose a learning-based framework for cooperative policies from decentralized observations using multi-agent reinforcement learning (MARL), supported by an architecture that handles variable numbers of humans and temporal dependencies.
  arXiv  Detail & Related papers  (2026-04-25T07:20:15Z)
• Event-Level Detection of Surgical Instrument Handovers in Videos with Interpretable Vision Models [32.17113044802459]
  We propose a vision framework for event-level detection and direction classification of instrument handovers in surgical videos.<n>A unified multi-task formulation jointly predicts handover occurrence and interaction direction.<n>Experiments on a dataset of kidney transplant procedures demonstrate strong performance.
  arXiv  Detail & Related papers  (2026-04-08T20:31:28Z)
• Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation [7.281396624646809]
  We introduce a gaze-regularized training framework that aligns VLA models' internal attention with human visual patterns.<n>When integrated into existing VLA architectures, our approach yields 4-12% improvements across manipulation benchmarks.
  arXiv  Detail & Related papers  (2026-03-24T13:50:28Z)
• MoE-ACT: Improving Surgical Imitation Learning Policies through Supervised Mixture-of-Experts [1.6646268910871171]
  We present a supervised Mixture-of-Experts architecture designed for phase-structured surgical manipulation tasks.<n>We show that a lightweight action decoder policy can learn complex, long-horizon manipulation from less than 150 demonstrations.<n>We present preliminary results of policy roll-outs during in vivo porcine surgery.
  arXiv  Detail & Related papers  (2026-01-29T16:50:14Z)
• \textsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation [50.027425808733994]
  textscNaVIDA is a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution.<n>textscNaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions.<n>Experiments show that textscNaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters.
  arXiv  Detail & Related papers  (2026-01-26T06:16:17Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• HAGI++: Head-Assisted Gaze Imputation and Generation [19.626054627997778]
  We introduce HAGI++ - a multi-modal diffusion-based approach for gaze data imputation.<n>It uses the integrated head orientation sensors to exploit the inherent correlation between head and eye movements.<n>Our method paves the way for more complete and accurate eye gaze recordings in real-world settings.
  arXiv  Detail & Related papers  (2025-11-04T10:51:34Z)
• Human Scanpath Prediction in Target-Present Visual Search with Semantic-Foveal Bayesian Attention [49.99728312519117]
  SemBA-FAST is a top-down framework designed for predicting human visual attention in target-present visual search.<n>We evaluate SemBA-FAST on the COCO-Search18 benchmark dataset, comparing its performance against other scanpath prediction models.<n>These findings provide valuable insights into the capabilities of semantic-foveal probabilistic frameworks for human-like attention modelling.
  arXiv  Detail & Related papers  (2025-07-24T15:19:23Z)
• Airway Skill Assessment with Spatiotemporal Attention Mechanisms Using Human Gaze [2.125763116614213]
  Airway management skills are critical in emergency medicine and are typically assessed through subjective evaluation.<n>This paper proposes a machine learning-based approach for assessing airway skills, specifically endotracheal intubation (ETI)<n>The proposed system leverages an attention mechanism guided by the human gaze to enhance the recognition of successful and unsuccessful ETI procedures.
  arXiv  Detail & Related papers  (2025-06-24T04:40:58Z)
• EndoARSS: Adapting Spatially-Aware Foundation Model for Efficient Activity Recognition and Semantic Segmentation in Endoscopic Surgery [11.286605039002419]
  Endoscopic surgery is the gold standard for robotic-assisted minimally invasive surgery.<n>Traditional deep learning models often struggle with cross-activity interference, leading to suboptimal performance in each downstream task.<n>We propose EndoARSS, a novel multi-task learning framework specifically designed for endoscopy surgery activity recognition and semantic segmentation.
  arXiv  Detail & Related papers  (2025-06-07T15:18:43Z)
• Understanding and Improving Training-Free AI-Generated Image Detections with Vision Foundation Models [68.90917438865078]
  Deepfake techniques for facial synthesis and editing pose serious risks for generative models.<n>In this paper, we investigate how detection performance varies across model backbones, types, and datasets.<n>We introduce Contrastive Blur, which enhances performance on facial images, and MINDER, which addresses noise type bias, balancing performance across domains.
  arXiv  Detail & Related papers  (2024-11-28T13:04:45Z)
• Towards Evaluating the Robustness of Visual State Space Models [63.14954591606638]
  Vision State Space Models (VSSMs) have demonstrated remarkable performance in visual perception tasks. However, their robustness under natural and adversarial perturbations remains a critical concern. We present a comprehensive evaluation of VSSMs' robustness under various perturbation scenarios.
  arXiv  Detail & Related papers  (2024-06-13T17:59:44Z)
• More Than Meets the Eye: Analyzing Anesthesiologists' Visual Attention in the Operating Room Using Deep Learning Models [0.0]
  Currently, most studies employ wearable eye-tracking technologies to analyze anesthesiologists' visual patterns. By utilizing a novel eye-tracking method in the form of deep learning models that process monitor-mounted webcams, we collected continuous behavioral data. We distinguished between baseline VA distribution during uneventful periods to patterns associated with active phases or during critical, unanticipated incidents.
  arXiv  Detail & Related papers  (2023-08-10T11:12:04Z)
• Weakly-Supervised Physically Unconstrained Gaze Estimation [80.66438763587904]
  We tackle the previously unexplored problem of weakly-supervised gaze estimation from videos of human interactions. We propose a training algorithm along with several novel loss functions especially designed for the task. We show significant improvements in (a) the accuracy of semi-supervised gaze estimation and (b) cross-domain generalization on the state-of-the-art physically unconstrained in-the-wild Gaze360 gaze estimation benchmark.
  arXiv  Detail & Related papers  (2021-05-20T14:58:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.