BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation

• URL: http://arxiv.org/abs/2407.17952v2
• Date: Wed, 6 Nov 2024 14:58:17 GMT
• Title: BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation
• Authors: Xiang Zhang, Bingxin Ke, Hayko Riemenschneider, Nando Metzger, Anton Obukhov, Markus Gross, Konrad Schindler, Christopher Schroers,
• Abstract summary: BetterDepth is a conditional diffusion-based refiner that takes the prediction from pre-trained MDE models as depth conditioning. BetterDepth achieves state-of-the-art zero-shot MDE performance on diverse public datasets and on in-the-wild scenes.
• Score: 25.047835960649167
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: By training over large-scale datasets, zero-shot monocular depth estimation (MDE) methods show robust performance in the wild but often suffer from insufficient detail. Although recent diffusion-based MDE approaches exhibit a superior ability to extract details, they struggle in geometrically complex scenes that challenge their geometry prior, trained on less diverse 3D data. To leverage the complementary merits of both worlds, we propose BetterDepth to achieve geometrically correct affine-invariant MDE while capturing fine details. Specifically, BetterDepth is a conditional diffusion-based refiner that takes the prediction from pre-trained MDE models as depth conditioning, in which the global depth layout is well-captured, and iteratively refines details based on the input image. For the training of such a refiner, we propose global pre-alignment and local patch masking methods to ensure BetterDepth remains faithful to the depth conditioning while learning to add fine-grained scene details. With efficient training on small-scale synthetic datasets, BetterDepth achieves state-of-the-art zero-shot MDE performance on diverse public datasets and on in-the-wild scenes. Moreover, BetterDepth can improve the performance of other MDE models in a plug-and-play manner without further re-training.

Related papers

• FiffDepth: Feed-forward Transformation of Diffusion-Based Generators for Detailed Depth Estimation [31.06080108012735]
  FiffDepth is a framework that transforms diffusion-based image generators into a feedforward architecture for detailed depth estimation. It achieves enhanced accuracy, stability, and fine-grained detail, offering a significant improvement in MDE performance across diverse real-world scenarios.
  arXiv  Detail & Related papers  (2024-12-01T04:59:34Z)
• IPoD: Implicit Field Learning with Point Diffusion for Generalizable 3D Object Reconstruction from Single RGB-D Images [50.4538089115248]
  Generalizable 3D object reconstruction from single-view RGB-D images remains a challenging task. We propose a novel approach, IPoD, which harmonizes implicit field learning with point diffusion. Experiments conducted on the CO3D-v2 dataset affirm the superiority of IPoD, achieving 7.8% improvement in F-score and 28.6% in Chamfer distance over existing methods.
  arXiv  Detail & Related papers  (2024-03-30T07:17:37Z)
• UniDepth: Universal Monocular Metric Depth Estimation [81.80512457953903]
  We propose a new model, UniDepth, capable of reconstructing metric 3D scenes from solely single images across domains. Our model exploits a pseudo-spherical output representation, which disentangles camera and depth representations. Thorough evaluations on ten datasets in a zero-shot regime consistently demonstrate the superior performance of UniDepth.
  arXiv  Detail & Related papers  (2024-03-27T18:06:31Z)
• SM4Depth: Seamless Monocular Metric Depth Estimation across Multiple Cameras and Scenes by One Model [72.0795843450604]
  Current approaches face challenges in maintaining consistent accuracy across diverse scenes. These methods rely on extensive datasets comprising millions, if not tens of millions, of data for training. This paper presents SM$4$Depth, a model that seamlessly works for both indoor and outdoor scenes.
  arXiv  Detail & Related papers  (2024-03-13T14:08:25Z)
• Robust Geometry-Preserving Depth Estimation Using Differentiable Rendering [93.94371335579321]
  We propose a learning framework that trains models to predict geometry-preserving depth without requiring extra data or annotations. Comprehensive experiments underscore our framework's superior generalization capabilities. Our innovative loss functions empower the model to autonomously recover domain-specific scale-and-shift coefficients.
  arXiv  Detail & Related papers  (2023-09-18T12:36:39Z)
• FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models [67.96827539201071]
  We propose a novel test-time optimization approach for 3D scene reconstruction. Our method achieves state-of-the-art cross-dataset reconstruction on five zero-shot testing datasets.
  arXiv  Detail & Related papers  (2023-08-10T17:55:02Z)
• Monocular Depth Estimation using Diffusion Models [39.27361388836347]
  We introduce innovations to address problems arising due to noisy, incomplete depth maps in training data. To cope with the limited availability of data for supervised training, we leverage pre-training on self-supervised image-to-image translation tasks. Our DepthGen model achieves SOTA performance on the indoor NYU dataset, and near SOTA results on the outdoor KITTI dataset.
  arXiv  Detail & Related papers  (2023-02-28T18:08:21Z)
• SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes [58.89295356901823]
  Self-supervised monocular depth estimation has shown impressive results in static scenes. It relies on the multi-view consistency assumption for training networks, however, that is violated in dynamic object regions. We introduce an external pretrained monocular depth estimation model for generating single-image depth prior. Our model can predict sharp and accurate depth maps, even when training from monocular videos of highly-dynamic scenes.
  arXiv  Detail & Related papers  (2022-11-07T16:17:47Z)
• Dense Depth Distillation with Out-of-Distribution Simulated Images [30.79756881887895]
  We study data-free knowledge distillation (KD) for monocular depth estimation (MDE) KD learns a lightweight model for real-world depth perception tasks by compressing it from a trained teacher model while lacking training data in the target domain. We show that our method outperforms the baseline KD by a good margin and even slightly better performance with as few as 1/6 of training images.
  arXiv  Detail & Related papers  (2022-08-26T07:10:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.