Foundation Models Meet Low-Cost Sensors: Test-Time Adaptation for Rescaling Disparity for Zero-Shot Metric Depth Estimation

• URL: http://arxiv.org/abs/2412.14103v1
• Date: Wed, 18 Dec 2024 17:50:15 GMT
• Title: Foundation Models Meet Low-Cost Sensors: Test-Time Adaptation for Rescaling Disparity for Zero-Shot Metric Depth Estimation
• Authors: Rémi Marsal, Alexandre Chapoutot, Philippe Xu, David Filliat,
• Abstract summary: We propose a new method to rescale Depth Anything predictions using 3D points provided by low-cost sensors or techniques such as low-resolution LiDAR. Our experiments highlight improvements relative to other metric depth estimation methods and competitive results compared to fine-tuned approaches.
• Score: 46.037640130193566
• License:
• Abstract: The recent development of foundation models for monocular depth estimation such as Depth Anything paved the way to zero-shot monocular depth estimation. Since it returns an affine-invariant disparity map, the favored technique to recover the metric depth consists in fine-tuning the model. However, this stage is costly to perform because of the training but also due to the creation of the dataset. It must contain images captured by the camera that will be used at test time and the corresponding ground truth. Moreover, the fine-tuning may also degrade the generalizing capacity of the original model. Instead, we propose in this paper a new method to rescale Depth Anything predictions using 3D points provided by low-cost sensors or techniques such as low-resolution LiDAR, stereo camera, structure-from-motion where poses are given by an IMU. Thus, this approach avoids fine-tuning and preserves the generalizing power of the original depth estimation model while being robust to the noise of the sensor or of the depth model. Our experiments highlight improvements relative to other metric depth estimation methods and competitive results compared to fine-tuned approaches. Code available at https://gitlab.ensta.fr/ssh/monocular-depth-rescaling.

Related papers

• Revisiting Gradient-based Uncertainty for Monocular Depth Estimation [10.502852645001882]
  We introduce gradient-based uncertainty estimation for monocular depth estimation models. We demonstrate that our approach is effective in determining the uncertainty without re-training. In particular, for models trained with monocular sequences and therefore most prone to uncertainty, our method outperforms related approaches.
  arXiv  Detail & Related papers  (2025-02-09T17:21:41Z)
• Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation [74.28509379811084]
  Metric3D v2 is a geometric foundation model for zero-shot metric depth and surface normal estimation from a single image. We propose solutions for both metric depth estimation and surface normal estimation. Our method enables the accurate recovery of metric 3D structures on randomly collected internet images.
  arXiv  Detail & Related papers  (2024-03-22T02:30:46Z)
• Metrically Scaled Monocular Depth Estimation through Sparse Priors for Underwater Robots [0.0]
  We formulate a deep learning model that fuses sparse depth measurements from triangulated features to improve the depth predictions. The network is trained in a supervised fashion on the forward-looking underwater dataset, FLSea. The method achieves real-time performance, running at 160 FPS on a laptop GPU and 7 FPS on a single CPU core.
  arXiv  Detail & Related papers  (2023-10-25T16:32:31Z)
• FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models [67.96827539201071]
  We propose a novel test-time optimization approach for 3D scene reconstruction. Our method achieves state-of-the-art cross-dataset reconstruction on five zero-shot testing datasets.
  arXiv  Detail & Related papers  (2023-08-10T17:55:02Z)
• Single Image Depth Prediction Made Better: A Multivariate Gaussian Take [163.14849753700682]
  We introduce an approach that performs continuous modeling of per-pixel depth. Our method's accuracy (named MG) is among the top on the KITTI depth-prediction benchmark leaderboard.
  arXiv  Detail & Related papers  (2023-03-31T16:01:03Z)
• Towards Accurate Reconstruction of 3D Scene Shape from A Single Monocular Image [91.71077190961688]
  We propose a two-stage framework that first predicts depth up to an unknown scale and shift from a single monocular image. We then exploits 3D point cloud data to predict the depth shift and the camera's focal length that allow us to recover 3D scene shapes. We test our depth model on nine unseen datasets and achieve state-of-the-art performance on zero-shot evaluation.
  arXiv  Detail & Related papers  (2022-08-28T16:20:14Z)
• Learning to Recover 3D Scene Shape from a Single Image [98.20106822614392]
  We propose a two-stage framework that first predicts depth up to an unknown scale and shift from a single monocular image. We then use 3D point cloud encoders to predict the missing depth shift and focal length that allow us to recover a realistic 3D scene shape.
  arXiv  Detail & Related papers  (2020-12-17T02:35:13Z)
• Variational Monocular Depth Estimation for Reliability Prediction [12.951621755732544]
  Self-supervised learning for monocular depth estimation is widely investigated as an alternative to supervised learning approach. Previous works have successfully improved the accuracy of depth estimation by modifying the model structure. In this paper, we theoretically formulate a variational model for the monocular depth estimation to predict the reliability of the estimated depth image.
  arXiv  Detail & Related papers  (2020-11-24T06:23:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.