DEFOM-Stereo: Depth Foundation Model Based Stereo Matching

• URL: http://arxiv.org/abs/2501.09466v1
• Date: Thu, 16 Jan 2025 10:59:29 GMT
• Title: DEFOM-Stereo: Depth Foundation Model Based Stereo Matching
• Authors: Hualie Jiang, Zhiqiang Lou, Laiyan Ding, Rui Xu, Minglang Tan, Wenjie Jiang, Rui Huang,
• Abstract summary: DEFOM-Stereo is built to facilitate robust stereo matching with monocular depth cues. DEFOM-Stereo is verified to have comparable performance on the Scene Flow dataset with state-of-the-art (SOTA) methods. Our model simultaneously outperforms previous models on the individual benchmarks.
• Score: 12.22373236061929
• License:
• Abstract: Stereo matching is a key technique for metric depth estimation in computer vision and robotics. Real-world challenges like occlusion and non-texture hinder accurate disparity estimation from binocular matching cues. Recently, monocular relative depth estimation has shown remarkable generalization using vision foundation models. Thus, to facilitate robust stereo matching with monocular depth cues, we incorporate a robust monocular relative depth model into the recurrent stereo-matching framework, building a new framework for depth foundation model-based stereo-matching, DEFOM-Stereo. In the feature extraction stage, we construct the combined context and matching feature encoder by integrating features from conventional CNNs and DEFOM. In the update stage, we use the depth predicted by DEFOM to initialize the recurrent disparity and introduce a scale update module to refine the disparity at the correct scale. DEFOM-Stereo is verified to have comparable performance on the Scene Flow dataset with state-of-the-art (SOTA) methods and notably shows much stronger zero-shot generalization. Moreover, DEFOM-Stereo achieves SOTA performance on the KITTI 2012, KITTI 2015, Middlebury, and ETH3D benchmarks, ranking 1st on many metrics. In the joint evaluation under the robust vision challenge, our model simultaneously outperforms previous models on the individual benchmarks. Both results demonstrate the outstanding capabilities of the proposed model.

Related papers

• MonoDINO-DETR: Depth-Enhanced Monocular 3D Object Detection Using a Vision Foundation Model [2.0624236247076397]
  This study employs a Vision Transformer (ViT)-based foundation model as the backbone, which excels at capturing global features for depth estimation. It integrates a detection transformer (DETR) architecture to improve both depth estimation and object detection performance in a one-stage manner. The proposed model outperforms recent state-of-the-art methods, as demonstrated through evaluations on the KITTI 3D benchmark and a custom dataset collected from high-elevation racing environments.
  arXiv  Detail & Related papers  (2025-02-01T04:37:13Z)
• FoundationStereo: Zero-Shot Stereo Matching [50.79202911274819]
  FoundationStereo is a foundation model for stereo depth estimation. We first construct a large-scale (1M stereo pairs) synthetic training dataset. We then design a number of network architecture components to enhance scalability.
  arXiv  Detail & Related papers  (2025-01-17T01:01:44Z)
• Unsupervised Monocular Depth Estimation Based on Hierarchical Feature-Guided Diffusion [21.939618694037108]
  Unsupervised monocular depth estimation has received widespread attention because of its capability to train without ground truth. We employ a well-converging diffusion model among generative networks for unsupervised monocular depth estimation. This model significantly enriches the model's capacity for learning and interpreting depth distribution.
  arXiv  Detail & Related papers  (2024-06-14T07:31:20Z)
• Stealing Stable Diffusion Prior for Robust Monocular Depth Estimation [33.140210057065644]
  This paper introduces a novel approach named Stealing Stable Diffusion (SSD) prior for robust monocular depth estimation. The approach addresses this limitation by utilizing stable diffusion to generate synthetic images that mimic challenging conditions. The effectiveness of the approach is evaluated on nuScenes and Oxford RobotCar, two challenging public datasets.
  arXiv  Detail & Related papers  (2024-03-08T05:06:31Z)
• PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation [47.53810786827547]
  Single image depth estimation is a foundational task in computer vision and generative modeling. We present PatchFusion, a novel tile-based framework with three key components to improve the current state of the art. Experiments on UnrealStereo4K, MVS- Synth, and Middleburry 2014 demonstrate that our framework can generate high-resolution depth maps with intricate details.
  arXiv  Detail & Related papers  (2023-12-04T19:03:12Z)
• DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation [50.08080424613603]
  Long-range correlation is essential for accurate monocular depth estimation. We propose to leverage the Transformer to model this global context with an effective attention mechanism. Our proposed model, termed DepthFormer, surpasses state-of-the-art monocular depth estimation methods with prominent margins.
  arXiv  Detail & Related papers  (2022-03-27T05:03:56Z)
• Stereo Neural Vernier Caliper [57.187088191829886]
  We propose a new object-centric framework for learning-based stereo 3D object detection. We tackle a problem of how to predict a refined update given an initial 3D cuboid guess. Our approach achieves state-of-the-art performance on the KITTI benchmark.
  arXiv  Detail & Related papers  (2022-03-21T14:36:07Z)
• SMD-Nets: Stereo Mixture Density Networks [68.56947049719936]
  We propose Stereo Mixture Density Networks (SMD-Nets), a simple yet effective learning framework compatible with a wide class of 2D and 3D architectures. Specifically, we exploit bimodal mixture densities as output representation and show that this allows for sharp and precise disparity estimates near discontinuities. We carry out comprehensive experiments on a new high-resolution and highly realistic synthetic stereo dataset, consisting of stereo pairs at 8Mpx resolution, as well as on real-world stereo datasets.
  arXiv  Detail & Related papers  (2021-04-08T16:15:46Z)
• Consistency Guided Scene Flow Estimation [159.24395181068218]
  CGSF is a self-supervised framework for the joint reconstruction of 3D scene structure and motion from stereo video. We show that the proposed model can reliably predict disparity and scene flow in challenging imagery. It achieves better generalization than the state-of-the-art, and adapts quickly and robustly to unseen domains.
  arXiv  Detail & Related papers  (2020-06-19T17:28:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.