DepthPilot: From Controllability to Interpretability in Colonoscopy Video Generation

• URL: http://arxiv.org/abs/2604.26232v1
• Date: Wed, 29 Apr 2026 02:24:28 GMT
• Title: DepthPilot: From Controllability to Interpretability in Colonoscopy Video Generation
• Abstract summary: We propose DepthPilot, the first interpretable framework for colonoscopy video generation.<n>To achieve explicit geometric grounding, DepthPilot devises a prior distribution alignment strategy.<n> depth constraints are injected into the backbone diffusion via parameter-efficient fine-tuning to ensure anatomical fidelity.
• Score: 22.17232304282025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Controllable medical video generation has achieved remarkable progress, but it still lacks interpretability, which requires the alignment of generated contents with physical priors and faithful clinical manifestations. To push the boundaries from mere controllability to interpretability, we propose DepthPilot, the first interpretable framework for colonoscopy video generation. This work takes a step toward trustworthy generation through two synergistic paradigms. To achieve explicit geometric grounding, DepthPilot devises a prior distribution alignment strategy, injecting depth constraints into the diffusion backbone via parameter-efficient fine-tuning to ensure anatomical fidelity. To enhance intrinsic nonlinear modeling under these geometric constraints, DepthPilot employs an adaptive spline denoising module, replacing fixed linear weights with learnable spline functions to capture complex spatio-temporal dynamics. Extensive evaluations across three public datasets and in-house clinical data confirm DepthPilot's robust ability to produce physically consistent videos. It achieves FID scores below 15 across all benchmarks and ranks first in clinician assessments, bridging the gap between "visually realistic" and "clinically interpretable". Moreover, DepthPilot-generated videos are expected to enable reliable 3D reconstruction, facilitating surgical navigation and blind region identification, and serve as a foundation toward the colorectal world model.

Related papers

• Robust 3D Alignment of Generative Reconstructions via Partial Monocular Observations [46.983222121901235]
  Generative-to-Partial Monocular Observational Alignment is a critical but under-explored challenge in computer vision.<n>We propose a training-free and interpretable geometric alignment framework that grounds generative 3D priors via a 3D similarity transformation.<n>Experiments demonstrate that our method achieves stable and accurate registration, substantially outperforming both classical geometric pipelines and state-of-the-art learning-based baselines.
  arXiv  Detail & Related papers  (2026-07-01T06:30:47Z)
• Geometry-Consistent Endoscopic Representations for Image-Guided Navigation via Structured Foundation Model Adaptation [7.175105815225476]
  We propose a unified framework for learning geometry-consistent and domain-robust image representations for monocular endoscopy.<n>We show improved geometric and semantic representation quality, leading to better performance on downstream navigation tasks.<n>These results support hierarchy-aware, geometry-guided adaptation as a practical approach for endoscopic representation learning.
  arXiv  Detail & Related papers  (2026-06-15T22:41:28Z)
• Analyzing and Improving Fine-grained Preference Optimization in Medical LVLMs [53.697403481143404]
  Large Vision-Language Models (LVLMs) have achieved strong performance across medical imaging tasks.<n>But they remain prone to factual inconsistencies, poor visual grounding, and misalignment with clinically meaningful feedback.
  arXiv  Detail & Related papers  (2026-06-10T18:35:36Z)
• VesselSim: learning 3D blood vessel segmentation without expert annotations [7.634402582162916]
  We propose VesselSim, a two-stage framework for universal 3D blood vessel segmentation.<n>A 3D U-Net is trained solely on synthetic data to generate 16,500 anatomically plausible 3D angiographic volumes.<n>VesselSim achieves competitive performance with state-of-the-art segmentation foundation models.
  arXiv  Detail & Related papers  (2026-05-25T19:04:18Z)
• ViPS: Video-informed Pose Spaces for Auto-Rigged Meshes [55.32681167870698]
  Video-informed Pose Spaces (ViPS) is a feed-forward framework that discovers the latent distribution of valid articulations for auto-rigged meshes.<n>ViPS transfers generative video priors into a universal distribution over a given rig parameterization.<n>Our evaluations show that ViPS matches the performance of state-of-the-art methods trained on synthetic artist-created 4D data in both plausibility and diversity.
  arXiv  Detail & Related papers  (2026-04-19T21:21:11Z)
• DVD: Deterministic Video Depth Estimation with Generative Priors [87.46576463137801]
  DVD is the first framework to adapt pre-trained video diffusion models into single-pass depth regressors.<n>DVD achieves state-of-the-art zero-shot performance across benchmarks.<n>We fully release our pipeline, providing the whole training suite for SOTA video depth estimation to benefit the open-source community.
  arXiv  Detail & Related papers  (2026-03-12T17:58:06Z)
• MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models [59.180043227905294]
  We propose MedPruner, a training-free and model-agnostic hierarchical token pruning framework for efficient 3D medical image understanding.<n>We show that MedPruner enables models such as MedGemma to maintain or even exceed their original performance while retaining fewer than 5% of visual tokens.
  arXiv  Detail & Related papers  (2026-03-12T07:37:00Z)
• Preoperative-to-intraoperative Liver Registration for Laparoscopic Surgery via Latent-Grounded Correspondence Constraints [51.7011449975586]
  Land-Reg is a deformable registration framework that learns latent-grounded 2D-3D landmark correspondences.<n>For rigid registration, Land-Reg embraces a Cross-modal Latent Alignment module.<n>An Uncertainty-enhanced Overlap Landmark Detector with similarity matching is proposed to robustly estimate explicit 2D-3D landmark correspondences.
  arXiv  Detail & Related papers  (2026-03-02T10:44:03Z)
• Extending 2D foundational DINOv3 representations to 3D segmentation of neonatal brain MR images [3.186130813218338]
  The global MRI volume is decomposed into non-overlapping 3D windows or sub-cubes, each processed via a separate decoding arm built upon frozen high-fidelity features.<n>The proposed approach achieves a Dice score of 0.65 for a single 3D window.
  arXiv  Detail & Related papers  (2026-02-27T12:16:21Z)
• GRAFNet: Multiscale Retinal Processing via Guided Cortical Attention Feedback for Enhancing Medical Image Polyp Segmentation [6.834321209531585]
  We propose GRAFNet, a biologically inspired architecture that emulates the hierarchical organisation of the human visual system.<n>GRAFNet integrates three key modules: (1) a Guided Asymmetric Attention Module (GAAM) that mimics orientation-tuned cortical neurones to polyp boundaries, (2) a MultiScale Retinal Module (MSRM) that replicates retinal ganglion cell pathways for parallel multi-feature analysis, and (3) a Guided Cortical Attention Feedback Module (GCAFM) that applies predictive coding for iterative refinement.
  arXiv  Detail & Related papers  (2026-02-15T17:29:37Z)
• EndoStreamDepth: Temporally Consistent Monocular Depth Estimation for Endoscopic Video Streams [6.300100115696222]
  This work presents EndoStreamDepth, a monocular depth estimation framework for endoscopic video streams.<n>It provides accurate depth maps with sharp anatomical boundaries for each frame, temporally consistent predictions across frames, and real-time throughput.
  arXiv  Detail & Related papers  (2025-12-20T00:53:30Z)
• BronchOpt : Vision-Based Pose Optimization with Fine-Tuned Foundation Models for Accurate Bronchoscopy Navigation [6.915058920280426]
  We propose a vision-based pose optimization framework for 2D-3D registration between intra-operative endoscopic views and pre-operative CT anatomy.<n>A fine-tuned modality- and domain-invariant encoder enables direct similarity between real endoscopic RGB frames and CT-rendered depth maps.<n>Our model achieves an average translational error of 2.65 mm and a rotational error of 0.19 rad, demonstrating accurate and stable localization.
  arXiv  Detail & Related papers  (2025-11-12T15:58:05Z)
• Landmark-Free Preoperative-to-Intraoperative Registration in Laparoscopic Liver Resection [50.388465935739376]
  Liver registration by overlaying preoperative 3D models onto intraoperative 2D frames can assist surgeons in perceiving the spatial anatomy of the liver clearly for a higher surgical success rate.<n>Existing registration methods rely heavily on anatomical landmark-based, which encounter two major limitations.<n>We propose a landmark-free preoperative-to-intraoperative registration framework utilizing effective self-supervised learning.
  arXiv  Detail & Related papers  (2025-04-21T14:55:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.