Structure-to-Image: Zero-Shot Depth Estimation in Colonoscopy via High-Fidelity Sim-to-Real Adaptation

• URL: http://arxiv.org/abs/2602.21740v1
• Date: Wed, 25 Feb 2026 09:51:53 GMT
• Title: Structure-to-Image: Zero-Shot Depth Estimation in Colonoscopy via High-Fidelity Sim-to-Real Adaptation
• Abstract summary: Monocular depth estimation (MDE) for colonoscopy is hampered by the domain gap between simulated and real-world images.<n>Existing image-to-image translation methods, which use depth as a posterior constraint, often produce structural distortions and specular highlights.<n>We propose a Structure-to-Image paradigm that transforms the depth map from a passive constraint into an active generative foundation.
• Score: 6.679573883606033
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Monocular depth estimation (MDE) for colonoscopy is hampered by the domain gap between simulated and real-world images. Existing image-to-image translation methods, which use depth as a posterior constraint, often produce structural distortions and specular highlights by failing to balance realism with structure consistency. To address this, we propose a Structure-to-Image paradigm that transforms the depth map from a passive constraint into an active generative foundation. We are the first to introduce phase congruency to colonoscopic domain adaptation and design a cross-level structure constraint to co-optimize geometric structures and fine-grained details like vascular textures. In zero-shot evaluations conducted on a publicly available phantom dataset, the MDE model that was fine-tuned on our generated data achieved a maximum reduction of 44.18% in RMSE compared to competing methods. Our code is available at https://github.com/YyangJJuan/PC-S2I.git.

Related papers

• Feed-Forward Hierarchical Gaussian Diffusion for Extreme CT Reconstruction [5.827215998326956]
  HiGDiff is a feed-forward hierarchical Gaussian diffusion framework that decomposes reconstruction both spatially and from structure to detail.<n> Experiments on three distinct CT benchmark datasets demonstrate state-of-the-art reconstruction performance across isolated, paired, and joint degradation settings.
  arXiv  Detail & Related papers  (2026-08-15T17:56:09Z)
• Spectral Rectification for Parameter-Efficient Adaptation of Foundation Models in Colonoscopy Depth Estimation [6.285139034905719]
  Foundation models trained on natural images fail to generalize directly to colonoscopy.<n>We propose SpecDepth, a parameter-efficient adaptation framework that preserves the robust geometric representations of the pre-trained models.<n>On the public C3VD and SimCol3D datasets, SpecDepth achieved state-of-the-art performance with an absolute relative error of 0.022 and 0.027.
  arXiv  Detail & Related papers  (2026-03-16T14:50:22Z)
• StarryGazer: Leveraging Monocular Depth Estimation Models for Domain-Agnostic Single Depth Image Completion [56.28564075246147]
  StarryGazer is a framework that predicts dense depth images from a single sparse depth image and an RGB image.<n>We employ a pre-trained MDE model to produce relative depth images.<n>A refinement network is trained with the synthetic pairs, incorporating the relative depth maps and RGB images to improve the model's accuracy and robustness.
  arXiv  Detail & Related papers  (2025-12-15T09:56:09Z)
• ControlVP: Interactive Geometric Refinement of AI-Generated Images with Consistent Vanishing Points [32.23473666846317]
  We propose ControlVP, a user-guided framework for correcting vanishing point inconsistencies in generated images.<n>Our approach extends a pre-trained diffusion model by incorporating structural guidance derived from building contours.<n>Our method enhances global geometric consistency while maintaining visual fidelity comparable to the baselines.
  arXiv  Detail & Related papers  (2025-12-08T12:38:11Z)
• PPS-Ctrl: Controllable Sim-to-Real Translation for Colonoscopy Depth Estimation [12.549599571284304]
  We propose a novel image-to-image translation framework that preserves structure while generating realistic textures from clinical data.<n>Our key innovation integrates Stable Diffusion with ControlNet, conditioned on a latent representation extracted from a Per-Pixel Shading (PPS) map.<n> Experiments show our approach produces more realistic translations and improves depth estimation over GAN-based MI-CycleGAN.
  arXiv  Detail & Related papers  (2025-04-23T19:28:58Z)
• Decompositional Neural Scene Reconstruction with Generative Diffusion Prior [64.71091831762214]
  Decompositional reconstruction of 3D scenes, with complete shapes and detailed texture, is intriguing for downstream applications.<n>Recent approaches incorporate semantic or geometric regularization to address this issue, but they suffer significant degradation in underconstrained areas.<n>We propose DP-Recon, which employs diffusion priors in the form of Score Distillation Sampling (SDS) to optimize the neural representation of each individual object under novel views.
  arXiv  Detail & Related papers  (2025-03-19T02:11:31Z)
• Structure-preserving Image Translation for Depth Estimation in Colonoscopy Video [1.0485739694839669]
  We propose a pipeline of structure-preserving synthetic-to-real (sim2real) image translation. This allows us to generate large quantities of realistic-looking synthetic images for supervised depth estimation. We also propose a dataset of hand-picked sequences from clinical colonoscopies to improve the image translation process.
  arXiv  Detail & Related papers  (2024-08-19T17:02:16Z)
• ToDER: Towards Colonoscopy Depth Estimation and Reconstruction with Geometry Constraint Adaptation [67.22294293695255]
  We propose a novel reconstruction pipeline with a bi-directional adaptation architecture named ToDER to get precise depth estimations. Experimental results demonstrate that our approach can precisely predict depth maps in both realistic and synthetic colonoscopy videos.
  arXiv  Detail & Related papers  (2024-07-23T14:24:26Z)
• Q-SLAM: Quadric Representations for Monocular SLAM [85.82697759049388]
  We reimagine volumetric representations through the lens of quadrics. We use quadric assumption to rectify noisy depth estimations from RGB inputs. We introduce a novel quadric-decomposed transformer to aggregate information across quadrics.
  arXiv  Detail & Related papers  (2024-03-12T23:27:30Z)
• DiAD: A Diffusion-based Framework for Multi-class Anomaly Detection [55.48770333927732]
  We propose a Difusion-based Anomaly Detection (DiAD) framework for multi-class anomaly detection. It consists of a pixel-space autoencoder, a latent-space Semantic-Guided (SG) network with a connection to the stable diffusion's denoising network, and a feature-space pre-trained feature extractor. Experiments on MVTec-AD and VisA datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-12-11T18:38:28Z)
• Improved Cryo-EM Pose Estimation and 3D Classification through Latent-Space Disentanglement [14.973360669658561]
  We propose a self-supervised variational autoencoder architecture called "HetACUMN" based on amortized inference. Results on simulated datasets show that HetACUMN generated more accurate conformational classifications than other amortized or non-amortized methods.
  arXiv  Detail & Related papers  (2023-08-09T13:41:30Z)
• A geometry-aware deep network for depth estimation in monocular endoscopy [17.425158094539462]
  The proposed method is extensively validated across different datasets and clinical images. The generalizability of the proposed method achieves mean RMSE values of 12.604 (T1-L1), 9.930 (T2-L2), and 13.893 (colon) on the ColonDepth dataset.
  arXiv  Detail & Related papers  (2023-04-20T11:59:32Z)
• A Structure-Guided Diffusion Model for Large-Hole Image Completion [85.61681358977266]
  We develop a structure-guided diffusion model to fill large holes in images. Our method achieves a superior or comparable visual quality compared to state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-11-18T18:59:01Z)
• Image-to-Height Domain Translation for Synthetic Aperture Sonar [3.2662392450935416]
  In this work, we focus on collection geometry with respect to isotropic and anisotropic textures. The low grazing angle of the collection geometry, combined with orientation of the sonar path relative to anisotropic texture, poses a significant challenge for image-alignment and other multi-view scene understanding frameworks.
  arXiv  Detail & Related papers  (2021-12-12T19:53:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.