DEFOM-Stereo: Depth Foundation Model Based Stereo Matching

• URL: http://arxiv.org/abs/2501.09466v3
• Date: Wed, 23 Apr 2025 10:38:09 GMT
• Title: DEFOM-Stereo: Depth Foundation Model Based Stereo Matching
• Authors: Hualie Jiang, Zhiqiang Lou, Laiyan Ding, Rui Xu, Minglang Tan, Wenjie Jiang, Rui Huang,
• Abstract summary: DEFOM-Stereo is built to facilitate robust stereo matching with monocular depth cues.<n>It is verified to have much stronger zero-shot generalization compared with SOTA methods.<n>Our model simultaneously outperforms previous models on the individual benchmarks.
• Score: 12.22373236061929
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Stereo matching is a key technique for metric depth estimation in computer vision and robotics. Real-world challenges like occlusion and non-texture hinder accurate disparity estimation from binocular matching cues. Recently, monocular relative depth estimation has shown remarkable generalization using vision foundation models. Thus, to facilitate robust stereo matching with monocular depth cues, we incorporate a robust monocular relative depth model into the recurrent stereo-matching framework, building a new framework for depth foundation model-based stereo-matching, DEFOM-Stereo. In the feature extraction stage, we construct the combined context and matching feature encoder by integrating features from conventional CNNs and DEFOM. In the update stage, we use the depth predicted by DEFOM to initialize the recurrent disparity and introduce a scale update module to refine the disparity at the correct scale. DEFOM-Stereo is verified to have much stronger zero-shot generalization compared with SOTA methods. Moreover, DEFOM-Stereo achieves top performance on the KITTI 2012, KITTI 2015, Middlebury, and ETH3D benchmarks, ranking $1^{st}$ on many metrics. In the joint evaluation under the robust vision challenge, our model simultaneously outperforms previous models on the individual benchmarks, further demonstrating its outstanding capabilities.

Related papers

• Boosting Omnidirectional Stereo Matching with a Pre-trained Depth Foundation Model [62.37493746544967]
  Camera-based setups offer a cost-effective option by using stereo depth estimation to generate dense, high-resolution depth maps. Existing omnidirectional stereo matching approaches achieve only limited depth accuracy across diverse environments. We present DFI-OmniStereo, a novel omnidirectional stereo matching method that leverages a large-scale pre-trained foundation model for relative monocular depth estimation.
  arXiv  Detail & Related papers  (2025-03-30T16:24:22Z)
• MonoDINO-DETR: Depth-Enhanced Monocular 3D Object Detection Using a Vision Foundation Model [2.0624236247076397]
  This study employs a Vision Transformer (ViT)-based foundation model as the backbone, which excels at capturing global features for depth estimation. It integrates a detection transformer (DETR) architecture to improve both depth estimation and object detection performance in a one-stage manner. The proposed model outperforms recent state-of-the-art methods, as demonstrated through evaluations on the KITTI 3D benchmark and a custom dataset collected from high-elevation racing environments.
  arXiv  Detail & Related papers  (2025-02-01T04:37:13Z)
• FoundationStereo: Zero-Shot Stereo Matching [50.79202911274819]
  FoundationStereo is a foundation model for stereo depth estimation. We first construct a large-scale (1M stereo pairs) synthetic training dataset. We then design a number of network architecture components to enhance scalability.
  arXiv  Detail & Related papers  (2025-01-17T01:01:44Z)
• A Simple yet Effective Test-Time Adaptation for Zero-Shot Monocular Metric Depth Estimation [46.037640130193566]
  We propose a new method to rescale Depth Anything predictions using 3D points provided by sensors or techniques such as low-resolution LiDAR or structure-from-motion with poses given by an IMU. Our experiments highlight enhancements relative to zero-shot monocular metric depth estimation methods, competitive results compared to fine-tuned approaches and a better robustness than depth completion approaches.
  arXiv  Detail & Related papers  (2024-12-18T17:50:15Z)
• Enhanced Encoder-Decoder Architecture for Accurate Monocular Depth Estimation [0.0]
  This paper introduces a novel deep learning-based approach using an enhanced encoder-decoder architecture, where the Inception-ResNet-v2 model serves as the encoder.<n> Experimental results on the NYU Depth V2 dataset show that our model achieves state-of-the-art performance, with an Absolute Relative Error (ARE) of 0.064, Root Mean Square Error (RMSE) of 0.228, and accuracy ($delta$ 1.25) of 89.3%.
  arXiv  Detail & Related papers  (2024-10-15T13:46:19Z)
• SR-Stereo & DAPE: Stepwise Regression and Pre-trained Edges for Practical Stereo Matching [2.8908326904081334]
  We propose a novel stepwise regression architecture to overcome domain discrepancies. To enhance the edge awareness of models adapting new domains with sparse ground truth, we propose Domain Adaptation based on Pre-trained Edges (DAPE) The proposed SR-Stereo and DAPE are extensively evaluated on SceneFlow, KITTI, Middbury 2014 and ETH3D.
  arXiv  Detail & Related papers  (2024-06-11T05:25:25Z)
• Stealing Stable Diffusion Prior for Robust Monocular Depth Estimation [33.140210057065644]
  This paper introduces a novel approach named Stealing Stable Diffusion (SSD) prior for robust monocular depth estimation. The approach addresses this limitation by utilizing stable diffusion to generate synthetic images that mimic challenging conditions. The effectiveness of the approach is evaluated on nuScenes and Oxford RobotCar, two challenging public datasets.
  arXiv  Detail & Related papers  (2024-03-08T05:06:31Z)
• PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation [47.53810786827547]
  Single image depth estimation is a foundational task in computer vision and generative modeling. We present PatchFusion, a novel tile-based framework with three key components to improve the current state of the art. Experiments on UnrealStereo4K, MVS- Synth, and Middleburry 2014 demonstrate that our framework can generate high-resolution depth maps with intricate details.
  arXiv  Detail & Related papers  (2023-12-04T19:03:12Z)
• Uncertainty Guided Adaptive Warping for Robust and Efficient Stereo Matching [77.133400999703]
  Correlation based stereo matching has achieved outstanding performance. Current methods with a fixed model do not work uniformly well across various datasets. This paper proposes a new perspective to dynamically calculate correlation for robust stereo matching.
  arXiv  Detail & Related papers  (2023-07-26T09:47:37Z)
• DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation [50.08080424613603]
  Long-range correlation is essential for accurate monocular depth estimation. We propose to leverage the Transformer to model this global context with an effective attention mechanism. Our proposed model, termed DepthFormer, surpasses state-of-the-art monocular depth estimation methods with prominent margins.
  arXiv  Detail & Related papers  (2022-03-27T05:03:56Z)
• Stereo Neural Vernier Caliper [57.187088191829886]
  We propose a new object-centric framework for learning-based stereo 3D object detection. We tackle a problem of how to predict a refined update given an initial 3D cuboid guess. Our approach achieves state-of-the-art performance on the KITTI benchmark.
  arXiv  Detail & Related papers  (2022-03-21T14:36:07Z)
• SMD-Nets: Stereo Mixture Density Networks [68.56947049719936]
  We propose Stereo Mixture Density Networks (SMD-Nets), a simple yet effective learning framework compatible with a wide class of 2D and 3D architectures. Specifically, we exploit bimodal mixture densities as output representation and show that this allows for sharp and precise disparity estimates near discontinuities. We carry out comprehensive experiments on a new high-resolution and highly realistic synthetic stereo dataset, consisting of stereo pairs at 8Mpx resolution, as well as on real-world stereo datasets.
  arXiv  Detail & Related papers  (2021-04-08T16:15:46Z)
• Consistency Guided Scene Flow Estimation [159.24395181068218]
  CGSF is a self-supervised framework for the joint reconstruction of 3D scene structure and motion from stereo video. We show that the proposed model can reliably predict disparity and scene flow in challenging imagery. It achieves better generalization than the state-of-the-art, and adapts quickly and robustly to unseen domains.
  arXiv  Detail & Related papers  (2020-06-19T17:28:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.