SutureAgent: Learning Surgical Trajectories via Goal-conditioned Offline RL in Pixel Space

• URL: http://arxiv.org/abs/2603.26720v1
• Date: Thu, 19 Mar 2026 01:36:07 GMT
• Title: SutureAgent: Learning Surgical Trajectories via Goal-conditioned Offline RL in Pixel Space
• Abstract summary: Predicting needle trajectories from endoscopic video is critical for robot-assisted suturing.<n>We formulate image-based needle trajectory prediction as a sequential decision-making problem.<n>SutureAgent encodes variable-length clips using an observation encoder to capture both local spatial cues and long-range temporal dynamics.
• Score: 22.300952176139948
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predicting surgical needle trajectories from endoscopic video is critical for robot-assisted suturing, enabling anticipatory planning, real-time guidance, and safer motion execution. Existing methods that directly learn motion distributions from visual observations tend to overlook the sequential dependency among adjacent motion steps. Moreover, sparse waypoint annotations often fail to provide sufficient supervision, further increasing the difficulty of supervised or imitation learning methods. To address these challenges, we formulate image-based needle trajectory prediction as a sequential decision-making problem, in which the needle tip is treated as an agent that moves step by step in pixel space. This formulation naturally captures the continuity of needle motion and enables the explicit modeling of physically plausible pixel-wise state transitions over time. From this perspective, we propose SutureAgent, a goal-conditioned offline reinforcement learning framework that leverages sparse annotations to dense reward signals via cubic spline interpolation, encouraging the policy to exploit limited expert guidance while exploring plausible future motion paths. SutureAgent encodes variable-length clips using an observation encoder to capture both local spatial cues and long-range temporal dynamics, and autoregressively predicts future waypoints through actions composed of discrete directions and continuous magnitudes. To enable stable offline policy optimization from expert demonstrations, we adopt Conservative Q-Learning with Behavioral Cloning regularization. Experiments on a new kidney wound suturing dataset containing 1,158 trajectories from 50 patients show that SutureAgent reduces Average Displacement Error by 58.6% compared with the strongest baseline, demonstrating the effectiveness of modeling needle trajectory prediction as pixel-level sequential action learning.

Related papers

• VidParse: Online Parsing of Egocentric Procedures Like a Pro [44.77721538280088]
  We present VidParse, an online framework that treats activity understanding as a graph-constrained inference problem.<n>By anchoring robust visual segments to hard procedural constraints, our approach preserves long-range state transitions and achieves up to a 10x improvement in complex multi-step parsing accuracy.
  arXiv  Detail & Related papers  (2026-08-27T18:00:02Z)
• From Spatial Semantics to Temporal Context: Leveraging Gaze Trajectory for Weakly Supervised Medical Image Segmentation [5.809537365593592]
  Eye tracking offers a cost-effective solution that can be naturally integrated into clinical.<n>We propose Trajectory-guided Uncertainty-aware Network (TrailNet)<n>TrailNet exploits gaze-supervised medical image segmentation from spatial semantics to temporal context by jointly leveraging fixations and trajectories.
  arXiv  Detail & Related papers  (2026-07-29T07:13:31Z)
• SegDiff: Segmented Trajectory Diffusion for Consistent and Adaptive Robot Manipulation [65.81476183180527]
  We introduce SegDiff, a closed-loop visuomotor policy that integrates the strengths of both paradigms.<n>SegDiff decomposes demonstrations into motion segments between keyposes and learns to predict the continuous trajectory from the current state to the next keypose.<n>SegDiff demonstrates significant performance gains over existing approaches across various simulated and real-world scenarios.
  arXiv  Detail & Related papers  (2026-07-13T02:48:03Z)
• MSACT: Multistage Spatial Alignment for Stable Low-Latency Fine Manipulation [4.439585594082787]
  Real-world fine manipulation, particularly in bimanual manipulation, requires low-latency control and stable visual localization.<n>Existing approaches make different trade-offs: action-chunking policies such as ACT enable low-latency execution and data efficiency but rely on dense visual features without explicit spatial consistency.<n>We introduce a multistage spatial attention module that extracts stable 2D attention points and jointly predicts future attention sequences with a temporal alignment loss.
  arXiv  Detail & Related papers  (2026-05-01T07:35:15Z)
• Referring-Aware Visuomotor Policy Learning for Closed-Loop Manipulation [91.20850436220267]
  We introduce the Referring-Aware Visuomotor Policy (ReV)<n>ReV incorporates sparse referring points provided by a human or a high-level reasoning planner.<n>It is trained only by applying targeted perturbations to expert demonstrations.
  arXiv  Detail & Related papers  (2026-04-07T07:41:11Z)
• Training-free Temporal Object Tracking in Surgical Videos [20.689670966095097]
  We present a novel approach for online object tracking in laparoscopic cholecystectomy (LC) surgical videos.<n>Our method addresses the challenges of costly pixel-level annotations and label inconsistencies inherent in existing datasets.
  arXiv  Detail & Related papers  (2026-03-08T23:09:16Z)
• GeoMotion: Rethinking Motion Segmentation via Latent 4D Geometry [61.24189040578178]
  We propose a fully learning-based approach that directly infers moving objects from latent feature representations via attention mechanisms.<n>Our key insight is to bypass explicit correspondence estimation and instead let the model learn to implicitly disentangle object and camera motion.<n>Our approach achieves state-of-the-art motion segmentation performance with high efficiency.
  arXiv  Detail & Related papers  (2026-02-25T11:36:33Z)
• ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving [64.42138266293202]
  ResAD is a Normalized Residual Trajectory Modeling framework.<n>It reframes the learning task to predict the residual deviation from an inertial reference.<n>On the NAVSIM benchmark, ResAD achieves a state-of-the-art PDMS of 88.6 using a vanilla diffusion policy.
  arXiv  Detail & Related papers  (2025-10-09T17:59:36Z)
• Fine-Grained Behavior and Lane Constraints Guided Trajectory Prediction Method [3.303114252531234]
  We present BLNet, a novel dualstream architecture that integrates behavioral intention recognition and lane constraint modeling.<n>Our network exhibits significant performance gains over existing direct regression and goal-based algorithms.
  arXiv  Detail & Related papers  (2025-03-27T13:06:57Z)
• Self-Supervised Class-Agnostic Motion Prediction with Spatial and Temporal Consistency Regularizations [53.797896854533384]
  Class-agnostic motion prediction methods directly predict the motion of the entire point cloud. While most existing methods rely on fully-supervised learning, the manual labeling of point cloud data is laborious and time-consuming. We introduce three simple spatial and temporal regularization losses, which facilitate the self-supervised training process effectively.
  arXiv  Detail & Related papers  (2024-03-20T02:58:45Z)
• Layout Sequence Prediction From Noisy Mobile Modality [53.49649231056857]
  Trajectory prediction plays a vital role in understanding pedestrian movement for applications such as autonomous driving and robotics. Current trajectory prediction models depend on long, complete, and accurately observed sequences from visual modalities. We propose LTrajDiff, a novel approach that treats objects obstructed or out of sight as equally important as those with fully visible trajectories.
  arXiv  Detail & Related papers  (2023-10-09T20:32:49Z)
• Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction [60.60223171143206]
  Trajectory prediction is a crucial undertaking in understanding entity movement or human behavior from observed sequences. Current methods often assume that the observed sequences are complete while ignoring the potential for missing values. This paper presents a unified framework, the Graph-based Conditional Variational Recurrent Neural Network (GC-VRNN), which can perform trajectory imputation and prediction simultaneously.
  arXiv  Detail & Related papers  (2023-03-28T14:27:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.