LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition

• URL: http://arxiv.org/abs/2607.19889v1
• Date: Wed, 22 Jul 2026 08:20:52 GMT
• Title: LAVIFT: Latent-Action-Guided Vision Fine-Tuning for Surgical Interaction Recognition
• Abstract summary: We introduce LAViFiT, an end-to-end latent-action-guided framework for vision-language fine-tuning.<n>An inverse dynamics model captures the visual changes induced by each action, while a forward world model drives the encoder to represent action-relevant regions.<n>A patch-level SIG Regularizer prevents local feature collapse without additional supervision, such as bounding boxes or pseudo-labels.
• Score: 13.912437629529691
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding instrument-tissue interactions is essential for context-aware surgical AI and autonomous robotic surgery. Pretrained vision-language models (VLMs) and vision encoders offer an alternative to conventional interaction classifiers by transferring broad visual and semantic knowledge. However, adapting them to fine-grained surgical interactions remains challenging: (1) freezing the vision encoder depends entirely on pretrained representations that may retain noise and provide weak spatial localization, while (2) full fine-tuning can improve global semantic alignment without ensuring that the encoder learns meaningful features in the correct action region. We address these limitations by introducing LAViFiT, an end-to-end latent-action-guided framework for vision-language fine-tuning. An inverse dynamics model captures the visual changes induced by each action, while a forward world model drives the encoder to represent action-relevant regions. A patch-level SIG Regularizer further prevents local feature collapse without additional supervision, such as bounding boxes or pseudo-labels. Experiments across multiple encoders and datasets improve recognition and image-text alignment, while representation analyses show stronger grounding over the complete instrument-tissue interaction region and more spatially coherent features.

Related papers

• IMPACT: Attention Is the Interaction Map for Scalable Interaction-Aware World Model Training [18.292611164084537]
  We develop IMPACT, a scalable Interaction-aware framework with Prior-guided Attention and Targeting.<n> IMPACT consistently outperforms the corresponding MSE-trained baselines, improving interaction fidelity, physical plausibility, and visual quality.
  arXiv  Detail & Related papers  (2026-08-31T18:00:36Z)
• SurgAM: Surgical Affordance Map Prediction with Multimodal Feature Fusion for Robot Autonomy [24.007662271891277]
  We introduce surgical affordance prediction, which identifies actionable regions for fundamental surgical actions from visual data.<n>A novel adaptive feature fusion framework is proposed that leverages the complementary strengths of a self-supervised vision transformer encoder.<n>A scene-guided attention decoder is proposed to focus on critical surgical areas while suppressing background distractions.
  arXiv  Detail & Related papers  (2026-07-05T16:12:01Z)
• TRUST: Efficient Abdominal Trauma Recognition via Image-to-Ultrasound-Video Transfer Learning [9.697220089288564]
  We propose a scan-aware PEIVTL framework that explicitly models fine-temporal variations to enable reliable ultrasound video understanding.<n>Experiments on in-house ultrasound trauma datasets demonstrate that TRUST outperforms state-of-the-art methods by 9.63% with superior computational efficiency.
  arXiv  Detail & Related papers  (2026-06-26T07:06:34Z)
• Making Foresight Actionable: Repurposing Representation Alignment in World Action Models [57.23863557252883]
  World Action Models (WAMs) offer a promising route for robot manipulation by using video generation models to model future scene evolution.<n> generating plausible visual futures does not always guarantee the extraction of accurate actions.<n>We propose AGRA, an Action-Grounded Representation Alignment objective.
  arXiv  Detail & Related papers  (2026-06-10T15:31:25Z)
• TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models [13.912437629529693]
  We propose TrajPred, a framework that encodes instrument trajectories to incorporate temporal motion cues.<n>We also incorporate prompt tuning and a verb-rephrasing technique to enable smooth adaptation to the instrument--tissue interaction recognition task.
  arXiv  Detail & Related papers  (2026-03-07T02:44:57Z)
• Revisiting Multi-Task Visual Representation Learning [52.93947931352643]
  We introduce MTV, a principled multi-task visual pretraining framework.<n>We leverage high-capacity "expert" models to synthesize dense, structured pseudo-labels at scale.<n>Our results demonstrate that MTV achieves "best-of-both-worlds" performance.
  arXiv  Detail & Related papers  (2026-01-20T11:59:19Z)
• GLip: A Global-Local Integrated Progressive Framework for Robust Visual Speech Recognition [72.29071664964633]
  We propose GLip, a Global-Local Integrated Progressive framework designed for robust visual speech recognition (VSR)<n>GLip learns to align both global and local visual features with corresponding acoustic speech units using easily accessible audio-visual data.<n>In the second stage, we introduce a Contextual Enhancement Module (CEM) to dynamically integrate local features with relevant global context.
  arXiv  Detail & Related papers  (2025-09-19T14:36:01Z)
• Emulating Human-like Adaptive Vision for Efficient and Flexible Machine Visual Perception [93.20637973889434]
  We introduce AdaptiveNN, a general framework aiming to drive a paradigm shift from 'passive' to 'active' vision models.<n> AdaptiveNN formulates visual perception as a coarse-to-fine sequential decision-making process.<n>We assess AdaptiveNN on 17 benchmarks spanning 9 tasks, including large-scale visual recognition, fine-grained discrimination, visual search, and processing images from real driving and medical scenarios.
  arXiv  Detail & Related papers  (2025-09-18T18:25:43Z)
• MASA: Motion-aware Masked Autoencoder with Semantic Alignment for Sign Language Recognition [94.56755080185732]
  We propose a Motion-Aware masked autoencoder with Semantic Alignment (MASA) that integrates rich motion cues and global semantic information. Our framework can simultaneously learn local motion cues and global semantic features for comprehensive sign language representation.
  arXiv  Detail & Related papers  (2024-05-31T08:06:05Z)
• Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment [60.31418655784291]
  We propose a novel unsupervised domain adaptation framework which can simultaneously transfer multi-modality knowledge, i.e., both kinematic and visual data, from simulator to real robot. It remedies the domain gap with enhanced transferable features by using temporal cues in videos, and inherent correlations in multi-modal towards recognizing gesture. Results show that our approach recovers the performance with great improvement gains, up to 12.91% in ACC and 20.16% in F1score without using any annotations in real robot.
  arXiv  Detail & Related papers  (2021-03-06T09:10:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.