BiliVLA: Scene-Aware Vision-Language-Action Model with Reinforcement Learning for Autonomous Biliary Endoscopic Navigation

• URL: http://arxiv.org/abs/2606.23531v2
• Date: Wed, 01 Jul 2026 05:58:33 GMT
• Title: BiliVLA: Scene-Aware Vision-Language-Action Model with Reinforcement Learning for Autonomous Biliary Endoscopic Navigation
• Abstract summary: Endoscopic retrograde cholangiopancreatography (ERCP) demands precise endoscopic navigation and stable biliary cannulation.<n>Recent robotic systems and vision-based assistance techniques improve operator ergonomics and provide perceptual cues.<n>We present BiliVLA, a scene-aware Vision-Language-Action framework that formulates biliary endoscopic navigation as an instruction-conditioned visuomotor learning problem.
• Score: 6.2182538853159
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Endoscopic retrograde cholangiopancreatography (ERCP) demands precise endoscopic navigation and stable biliary cannulation within a narrow monocular field characterized by specular reflections, partial occlusions, and frequent tissue contact. Although recent robotic systems and vision-based assistance techniques improve operator ergonomics and provide perceptual cues, their performance degrades under pronounced anatomical variability and safety-critical visual artifacts, which hinders reliable autonomy in cannulation-grade procedures. Here, we present BiliVLA, a scene-aware Vision-Language-Action (VLA) framework that formulates biliary endoscopic navigation as an instruction-conditioned visuomotor learning problem. Given an endoscopic observation and a stage-specific language instruction, BiliVLA jointly predicts the target category, a grounded bounding box, and a discrete three degrees of freedom (DoF) motor command for a continuum endoscope. The proposed framework incorporates scene-aware supervision to enhance semantic target consistency and safety-aware recovery supervision to induce conservative retreat behaviors under luminal wall contact. A key component of BiliVLA is a two-stage training paradigm that combines grounding-enhanced supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO), which significantly improves action reliability and decision consistency during closed-loop navigation. Across three ERCP subtasks, BiliVLA achieves an average action precision of 91.96\% and an overall success rate (SR) of 84.85\% in real-world phantom experiments. These results indicate that integrating semantic grounding, scene-aware learning, and reward-guided optimization improves perception-action alignment and enables robust autonomous endoscopic navigation.

Related papers

• Supervised Cross-Modal Feature Alignment for Zero-Wearable Freezing of Gait Detection in Parkinsonism [0.0]
  Objective assessment of Freezing of Gait (FoG) in Parkinson's disease relies predominantly on wearable Inertial Measurement Units (IMUs)<n>We propose a supervised cross-modal subspace distillation framework to resolve these physical observation limits.<n>During optimisation, pre-trained kinematic data from IMU sensors and contextual clinical metadata act as oracles to guide a deployable visual architecture.
  arXiv  Detail & Related papers  (2026-09-08T06:40:50Z)
• Partial Skeleton Visibility for Action Recognition: A Constrained Field-of-View Approach [60.596944437968524]
  PartialVisGraph is a novel framework for robust skeleton-based action recognition under constrained field-of-view.<n>It consistently achieves state-of-the-art accuracy under partial visibility, with gains of up to 68.8% on subsets with severe FoV restrictions.<n>Our approach offers a principled and practical pathway toward deployable skeleton-based action understanding in unconstrained environments.
  arXiv  Detail & Related papers  (2026-07-01T10:03:11Z)
• Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs [66.15429821530503]
  Persistent Visual Memory is a lightweight learnable module designed to strengthen sustained, on-demand access to visual evidence.<n>Experiments on Qwen3-VL models demonstrate that PVM brings notable improvements with negligible parameter overhead.<n>In-depth analysis reveals that PVM shows improved robustness in longer generations and accelerates internal prediction convergence.
  arXiv  Detail & Related papers  (2026-05-01T17:54:37Z)
• PromptCD: Test-Time Behavior Enhancement via Polarity-Prompt Contrastive Decoding [85.22047087898311]
  We introduce Polarity-Prompt Contrastive Decoding (PromptCD), a test-time behavior control method that generalizes contrastive decoding to broader enhancement settings.<n>PromptCD constructs paired positive and negative guiding prompts for a target behavior and contrasts model responses to reinforce desirable outcomes.<n>Experiments on the "3H" alignment objectives demonstrate consistent and substantial improvements, indicating that post-trained models can achieve meaningful self-enhancement purely at test time.
  arXiv  Detail & Related papers  (2026-02-24T08:56:52Z)
• Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining [15.995867664955348]
  We present a strategy-grounded framework that couples high-level vision-language inference with low-level closed-loop control.<n> offline, raw surgical videos are parsed into camera-relevant temporal events and structured as attributed event graphs.<n>Online, a fine-tuned Vision-Language Model (VLM) processes the live laparoscopic view to predict the dominant strategy and discrete image-based motion commands.
  arXiv  Detail & Related papers  (2026-02-24T02:56:39Z)
• Sample-Efficient Learning with Online Expert Correction for Autonomous Catheter Steering in Endovascular Bifurcation Navigation [9.822810178672782]
  Reinforcement learning (RL) has emerged as a promising approach for autonomous catheter steering.<n>This paper introduces a sample-efficient RL framework with online expert correction for autonomous catheter steering.
  arXiv  Detail & Related papers  (2026-02-23T07:58:08Z)
• Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs [67.69730908817321]
  Internal Positive Attention Dynamics (PAD) in LVLMs naturally reveal semantically core visual regions under the distortions of attention sinks.<n>We propose Positive Attention Dynamics Enhancement (PADE), a training-free attention intervention that constructs a PAD map to identify semantically core visual regions.
  arXiv  Detail & Related papers  (2026-02-17T13:08:06Z)
• \textsc{NaVIDA}: Vision-Language Navigation with Inverse Dynamics Augmentation [50.027425808733994]
  textscNaVIDA is a unified VLN framework that couples policy learning with action-grounded visual dynamics and adaptive execution.<n>textscNaVIDA augments training with chunk-based inverse-dynamics supervision to learn causal relationship between visual changes and corresponding actions.<n>Experiments show that textscNaVIDA achieves superior navigation performance compared to state-of-the-art methods with fewer parameters.
  arXiv  Detail & Related papers  (2026-01-26T06:16:17Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• Towards Autonomous Navigation in Endovascular Interventions [0.7100795934780609]
  This thesis presents an integrated AI-driven framework for autonomous guidewire navigation in complex vascular environments.<n>A high-fidelity, real-time simulation platform, CathSim, is introduced for reinforcement learning based catheter navigation.<n>The Expert Navigation Network is developed, a policy that fuses visual, kinematic, and force feedback for autonomous tool control.
  arXiv  Detail & Related papers  (2025-12-19T21:38:52Z)
• DINO-CVA: A Multimodal Goal-Conditioned Vision-to-Action Model for Autonomous Catheter Navigation [0.33727511459109777]
  This work moves towards autonomous catheter navigation by introducing DINO-CVA, a multimodal goal-conditioned behavior cloning framework.<n>The proposed model fuses visual observations and joystick kinematics into a joint embedding space, enabling policies that are both vision-aware and kinematic-aware.<n>Results show that DINO-CVA achieves high accuracy in predicting actions, matching the performance of a kinematics-only baseline.
  arXiv  Detail & Related papers  (2025-10-19T22:59:32Z)
• Uncertainty-Driven Expert Control: Enhancing the Reliability of Medical Vision-Language Models [52.2001050216955]
  Existing methods aim to enhance the performance of Medical Vision Language Model (MedVLM) by adjusting model structure, fine-tuning with high-quality data, or through preference fine-tuning.<n>We propose an expert-in-the-loop framework named Expert-Controlled-Free Guidance (Expert-CFG) to align MedVLM with clinical expertise without additional training.
  arXiv  Detail & Related papers  (2025-07-12T09:03:30Z)
• EndoVLA: Dual-Phase Vision-Language-Action Model for Autonomous Tracking in Endoscopy [26.132684811981143]
  Vision-Language-Action (VLA) models integrate visual perception, language grounding, and motion planning within an end-to-end framework.<n>EndoVLA performs three core tasks: (1) polyp tracking, (2) delineation and following of abnormal mucosal regions, and (3) adherence to circular markers during circumferential cutting.
  arXiv  Detail & Related papers  (2025-05-21T07:35:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.