RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions

• URL: http://arxiv.org/abs/2410.02924v2
• Date: Sun, 3 Nov 2024 02:20:28 GMT
• Title: RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions
• Abstract summary: Inferring depth from a single image is an ill-posed problem due to the loss of scale from perspective projection. Our goal is to recover metric-scaled depth maps through a linear transformation. Our method, RSA, takes as input a text caption describing objects present in an image and outputs the parameters of a linear transformation.
• Score: 47.614203035800735
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a method for metric-scale monocular depth estimation. Inferring depth from a single image is an ill-posed problem due to the loss of scale from perspective projection during the image formation process. Any scale chosen is a bias, typically stemming from training on a dataset; hence, existing works have instead opted to use relative (normalized, inverse) depth. Our goal is to recover metric-scaled depth maps through a linear transformation. The crux of our method lies in the observation that certain objects (e.g., cars, trees, street signs) are typically found or associated with certain types of scenes (e.g., outdoor). We explore whether language descriptions can be used to transform relative depth predictions to those in metric scale. Our method, RSA, takes as input a text caption describing objects present in an image and outputs the parameters of a linear transformation which can be applied globally to a relative depth map to yield metric-scaled depth predictions. We demonstrate our method on recent general-purpose monocular depth models on indoors (NYUv2, VOID) and outdoors (KITTI). When trained on multiple datasets, RSA can serve as a general alignment module in zero-shot settings. Our method improves over common practices in aligning relative to metric depth and results in predictions that are comparable to an upper bound of fitting relative depth to ground truth via a linear transformation.

Related papers

• Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth [36.90157943436845]
  We propose a framework for evaluating monocular depth estimators (MDEs) in the context of downstream tasks.<n>By combining depth predictions with feature correspondences in depth-aware geometric solvers, we use relative camera pose estimation accuracy as a proxy for depth quality.<n>Our framework can be applied to scenes where ground-truth depth is difficult to obtain.
  arXiv  Detail & Related papers  (2026-05-19T12:59:11Z)
• Learning Image-Adaptive Scale Fields for Metric Depth Recovery [13.22167540260594]
  We formulate metric depth recovery as image-adaptive scale field modeling.<n>Instead of directly correcting the depth, we reformulate the correction as a low-dimensional linear combination of image-adaptive basis maps.<n>This formulation yields improved metric depth accuracy, strong robustness under extreme anchor sparsity, and an interpretable decomposition of spatial scale variations.
  arXiv  Detail & Related papers  (2026-05-08T08:13:02Z)
• AnchorD: Metric Grounding of Monocular Depth Using Factor Graphs [16.622717763906262]
  Currently available depth sensors are prone to errors on transparent, specular, and general non-Lambertian surfaces.<n>We propose a training-free depth grounding framework that anchors monocular depth estimation priors from a depth foundation model.<n>Our method performs a patch-wise affine alignment, locally grounding monocular predictions in metric real-world depth.
  arXiv  Detail & Related papers  (2026-05-04T14:48:52Z)
• TR2M: Transferring Monocular Relative Depth to Metric Depth with Language Descriptions and Scale-Oriented Contrast [7.127920563966129]
  Current monocular depth estimation methods are mainly divided into metric depth estimation (MMDE) and relative depth estimation (MRDE)<n>MMDEs estimate depth in metric scale but are often limited to a specific domain. MRDEs generalize well across different domains, but with uncertain scales which hinders downstream applications.<n>Our approach, TR2M, utilizes both text description and image as inputs and estimates two rescale maps to transfer relative depth to metric depth at pixel level.
  arXiv  Detail & Related papers  (2025-06-16T11:50:00Z)
• VGLD: Visually-Guided Linguistic Disambiguation for Monocular Depth Scale Recovery [2.8834278113855896]
  VGLD (Visually-Guided Linguistic Disambiguation) is a framework that incorporates high-level visual semantics to resolve ambiguity in textual inputs.<n>By jointly encoding both image and text, VGLD predicts a set of global linear transformation parameters that align relative depth maps with metric scale.<n>Results show that VGLD significantly mitigates scale estimation bias caused by inconsistent or ambiguous language, achieving robust and accurate metric predictions.
  arXiv  Detail & Related papers  (2025-05-05T14:57:16Z)
• ScaleDepth: Decomposing Metric Depth Estimation into Scale Prediction and Relative Depth Estimation [62.600382533322325]
  We propose a novel monocular depth estimation method called ScaleDepth. Our method decomposes metric depth into scene scale and relative depth, and predicts them through a semantic-aware scale prediction module. Our method achieves metric depth estimation for both indoor and outdoor scenes in a unified framework.
  arXiv  Detail & Related papers  (2024-07-11T05:11:56Z)
• Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation [74.28509379811084]
  Metric3D v2 is a geometric foundation model for zero-shot metric depth and surface normal estimation from a single image. We propose solutions for both metric depth estimation and surface normal estimation. Our method enables the accurate recovery of metric 3D structures on randomly collected internet images.
  arXiv  Detail & Related papers  (2024-03-22T02:30:46Z)
• Metrically Scaled Monocular Depth Estimation through Sparse Priors for Underwater Robots [0.0]
  We formulate a deep learning model that fuses sparse depth measurements from triangulated features to improve the depth predictions. The network is trained in a supervised fashion on the forward-looking underwater dataset, FLSea. The method achieves real-time performance, running at 160 FPS on a laptop GPU and 7 FPS on a single CPU core.
  arXiv  Detail & Related papers  (2023-10-25T16:32:31Z)
• Towards Accurate Reconstruction of 3D Scene Shape from A Single Monocular Image [91.71077190961688]
  We propose a two-stage framework that first predicts depth up to an unknown scale and shift from a single monocular image. We then exploits 3D point cloud data to predict the depth shift and the camera's focal length that allow us to recover 3D scene shapes. We test our depth model on nine unseen datasets and achieve state-of-the-art performance on zero-shot evaluation.
  arXiv  Detail & Related papers  (2022-08-28T16:20:14Z)
• Virtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth Prediction [87.08227378010874]
  We show the importance of the high-order 3D geometric constraints for depth prediction. By designing a loss term that enforces a simple geometric constraint, we significantly improve the accuracy and robustness of monocular depth estimation. We show state-of-the-art results of learning metric depth on NYU Depth-V2 and KITTI.
  arXiv  Detail & Related papers  (2021-03-07T00:08:21Z)
• DiverseDepth: Affine-invariant Depth Prediction Using Diverse Data [110.29043712400912]
  We present a method for depth estimation with monocular images, which can predict high-quality depth on diverse scenes up to an affine transformation. Experiments show that our method outperforms previous methods on 8 datasets by a large margin with the zero-shot test setting.
  arXiv  Detail & Related papers  (2020-02-03T05:38:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.