Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation

• URL: http://arxiv.org/abs/2412.14015v1
• Date: Wed, 18 Dec 2024 16:32:12 GMT
• Title: Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation
• Authors: Haotong Lin, Sida Peng, Jingxiao Chen, Songyou Peng, Jiaming Sun, Minghuan Liu, Hujun Bao, Jiashi Feng, Xiaowei Zhou, Bingyi Kang,
• Abstract summary: We introduce prompting into depth foundation models, creating a new paradigm for metric depth estimation termed Prompt Depth Anything. We use a low-cost LiDAR as the prompt to guide the Depth Anything model for accurate metric depth output, achieving up to 4K resolution.
• Score: 108.04354143020886
• License:
• Abstract: Prompts play a critical role in unleashing the power of language and vision foundation models for specific tasks. For the first time, we introduce prompting into depth foundation models, creating a new paradigm for metric depth estimation termed Prompt Depth Anything. Specifically, we use a low-cost LiDAR as the prompt to guide the Depth Anything model for accurate metric depth output, achieving up to 4K resolution. Our approach centers on a concise prompt fusion design that integrates the LiDAR at multiple scales within the depth decoder. To address training challenges posed by limited datasets containing both LiDAR depth and precise GT depth, we propose a scalable data pipeline that includes synthetic data LiDAR simulation and real data pseudo GT depth generation. Our approach sets new state-of-the-arts on the ARKitScenes and ScanNet++ datasets and benefits downstream applications, including 3D reconstruction and generalized robotic grasping.

Related papers

• Revisiting Monocular 3D Object Detection from Scene-Level Depth Retargeting to Instance-Level Spatial Refinement [44.4805861813093]
  Monocular 3D object detection is challenging due to the lack of accurate depth. Existing depth-assisted solutions still exhibit inferior performance. We propose a novel depth-adapted monocular 3D object detection network.
  arXiv  Detail & Related papers  (2024-12-26T10:51:50Z)
• DepthLab: From Partial to Complete [80.58276388743306]
  Missing values remain a common challenge for depth data across its wide range of applications. This work bridges this gap with DepthLab, a foundation depth inpainting model powered by image diffusion priors. Our approach proves its worth in various downstream tasks, including 3D scene inpainting, text-to-3D scene generation, sparse-view reconstruction with DUST3R, and LiDAR depth completion.
  arXiv  Detail & Related papers  (2024-12-24T04:16:38Z)
• Robust Geometry-Preserving Depth Estimation Using Differentiable Rendering [93.94371335579321]
  We propose a learning framework that trains models to predict geometry-preserving depth without requiring extra data or annotations. Comprehensive experiments underscore our framework's superior generalization capabilities. Our innovative loss functions empower the model to autonomously recover domain-specific scale-and-shift coefficients.
  arXiv  Detail & Related papers  (2023-09-18T12:36:39Z)
• LiDAR Meta Depth Completion [47.99004789132264]
  We propose a meta depth completion network that uses data patterns to learn a task network to solve a given depth completion task effectively. While using a single model, our method yields significantly better results than a non-adaptive baseline trained on different LiDAR patterns. These advantages allow flexible deployment of a single depth completion model on different sensors.
  arXiv  Detail & Related papers  (2023-07-24T13:05:36Z)
• Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction [91.43066633305662]
  We propose a novel multi-task and multi-modal filtered transformer (MMFT) network for RGB-D salient object detection (SOD) Specifically, we unify three complementary tasks: depth estimation, salient object detection and contour estimation. The multi-task mechanism promotes the model to learn the task-aware features from the auxiliary tasks. Experiments show that it not only significantly surpasses the depth-based RGB-D SOD methods on multiple datasets, but also precisely predicts a high-quality depth map and salient contour at the same time.
  arXiv  Detail & Related papers  (2022-03-09T17:20:18Z)
• Consistent Depth Prediction under Various Illuminations using Dilated Cross Attention [1.332560004325655]
  We propose to use internet 3D indoor scenes and manually tune their illuminations to render photo-realistic RGB photos and their corresponding depth and BRDF maps. We perform cross attention on these dilated features to retain the consistency of depth prediction under different illuminations. Our method is evaluated by comparing it with current state-of-the-art methods on Vari dataset and a significant improvement is observed in experiments.
  arXiv  Detail & Related papers  (2021-12-15T10:02:46Z)
• DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network [3.1447111126464997]
  We propose DenseLiDAR, a novel real-time pseudo-depth guided depth completion neural network. We exploit dense pseudo-depth map obtained from simple morphological operations to guide the network. Our model is able to achieve the state-of-the-art performance at the highest frame rate of 50Hz.
  arXiv  Detail & Related papers  (2021-08-28T14:18:29Z)
• Virtual Normal: Enforcing Geometric Constraints for Accurate and Robust Depth Prediction [87.08227378010874]
  We show the importance of the high-order 3D geometric constraints for depth prediction. By designing a loss term that enforces a simple geometric constraint, we significantly improve the accuracy and robustness of monocular depth estimation. We show state-of-the-art results of learning metric depth on NYU Depth-V2 and KITTI.
  arXiv  Detail & Related papers  (2021-03-07T00:08:21Z)
• ADAADepth: Adapting Data Augmentation and Attention for Self-Supervised Monocular Depth Estimation [8.827921242078881]
  We propose ADAA, utilising depth augmentation as depth supervision for learning accurate and robust depth. We propose a relational self-attention module that learns rich contextual features and further enhances depth results. We evaluate our predicted depth on the KITTI driving dataset and achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2021-03-01T09:06:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.