Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

• URL: http://arxiv.org/abs/2609.08084v1
• Date: Tue, 08 Sep 2026 00:52:01 GMT
• Title: Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation
• Abstract summary: We revisit Marigold, a set of techniques for repurposing modern image generation and editing models, into state-of-the-art monocular depth estimators.<n>Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run.<n>The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D.
• Score: 68.73567966621631
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps. In this paper, we revisit Marigold, a set of techniques for repurposing modern image generation and editing models, powered by the diffusion transformer (DiT) architecture, into state-of-the-art monocular depth estimators. Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run. We analyze the artifacts of naive training and identify two effective remedies: aligning the model's internal representations with semantic features extracted from ground-truth, and adopting a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D. Qualitatively, our model resolves fur, foliage, and hair-thin edges that have eluded prior models. Furthermore, Marigold V2 achieves state-of-the-art results when applied to other dense regression tasks, such as surface normals estimation and intrinsic image decomposition. Project website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web

Related papers

• Re-Depth Anything: Test-Time Depth Refinement via Self-Supervised Re-lighting [14.53203375098443]
  We introduce Re-Depth Anything, a test-time self-supervision framework.<n>We fuse DA-V2 with the powerful priors of large-scale 2D diffusion models.<n>Our method performs label-free refinement directly on the input image.
  arXiv  Detail & Related papers  (2025-12-19T18:59:56Z)
• From Editor to Dense Geometry Estimator [77.21804448599009]
  We introduce textbfFE2E, a framework that adapts an advanced editing model based on Diffusion Transformer (DiT) architecture for dense geometry prediction.<n>FE2E achieves over 35% performance gains on the ETH3D dataset and outperforms the DepthAnything series, which is trained on 100$times$ data.
  arXiv  Detail & Related papers  (2025-09-04T15:58:50Z)
• Exploring Diffusion Transformer Designs via Grafting [82.91123758506876]
  We present grafting, a simple approach for editing pretrained diffusion transformers (DiTs) to materialize new architectures under small compute budgets.<n>We show that new diffusion model designs can be explored by grafting pretrained DiTs, with edits ranging from operator replacement to architecture restructuring.
  arXiv  Detail & Related papers  (2025-06-05T17:59:40Z)
• Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis [28.198808978074126]
  We present Marigold, a family of conditional generative models and a fine-tuning protocol that extracts the knowledge from pretrained latent diffusion models.<n>Marigold requires minimal modification of the pre-trained latent diffusion model's architecture, trains with small synthetic datasets on a single GPU over a few days, and demonstrates state-of-the-art zero-shot generalization.
  arXiv  Detail & Related papers  (2025-05-14T13:07:03Z)
• RI3D: Few-Shot Gaussian Splatting With Repair and Inpainting Diffusion Priors [13.883695200241524]
  RI3D is a novel approach that harnesses the power of diffusion models to reconstruct high-quality novel views given a sparse set of input images.<n>Our key contribution is separating the view synthesis process into two tasks of reconstructing visible regions and hallucinating missing regions.<n>We produce results with detailed textures in both visible and missing regions that outperform state-of-the-art approaches on a diverse set of scenes.
  arXiv  Detail & Related papers  (2025-03-13T20:16:58Z)
• UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler [62.06785782635153]
  We propose a new model, UniDepthV2, capable of reconstructing metric 3D scenes from solely single images across domains.<n>UniDepthV2 directly predicts metric 3D points from the input image at inference time without any additional information.<n>Our model exploits a pseudo-spherical output representation, which disentangles the camera and depth representations.
  arXiv  Detail & Related papers  (2025-02-27T14:03:15Z)
• DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features [65.8738034806085]
  DistillNeRF is a self-supervised learning framework for understanding 3D environments in autonomous driving scenes. Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs.
  arXiv  Detail & Related papers  (2024-06-17T21:15:13Z)
• Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation [20.230238670888454]
  We introduce Marigold, a method for affine-invariant monocular depth estimation. It can be fine-tuned in a couple of days on a single GPU using only synthetic training data. It delivers state-of-the-art performance across a wide range of datasets, including over 20% performance gains in specific cases.
  arXiv  Detail & Related papers  (2023-12-04T18:59:13Z)
• LiP-Flow: Learning Inference-time Priors for Codec Avatars via Normalizing Flows in Latent Space [90.74976459491303]
  We introduce a prior model that is conditioned on the runtime inputs and tie this prior space to the 3D face model via a normalizing flow in the latent space. A normalizing flow bridges the two representation spaces and transforms latent samples from one domain to another, allowing us to define a latent likelihood objective. We show that our approach leads to an expressive and effective prior, capturing facial dynamics and subtle expressions better.
  arXiv  Detail & Related papers  (2022-03-15T13:22:57Z)
• Bayesian Image Reconstruction using Deep Generative Models [7.012708932320081]
  In this work, we leverage state-of-the-art (SOTA) generative models for building powerful image priors. Our method, called Bayesian Reconstruction through Generative Models (BRGM), uses a single pre-trained generator model to solve different image restoration tasks.
  arXiv  Detail & Related papers  (2020-12-08T17:11:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.