Vision-Language Embodiment for Monocular Depth Estimation

• URL: http://arxiv.org/abs/2503.16535v2
• Date: Sat, 12 Apr 2025 03:26:26 GMT
• Title: Vision-Language Embodiment for Monocular Depth Estimation
• Authors: Jinchang Zhang, Guoyu Lu,
• Abstract summary: Current depth estimation models rely on inter-image relationships for supervised training.<n>We propose a method that embodies the camera model and its physical characteristics into a deep learning model.<n>The model can calculate embodied scene depth in real-time based on immediate environmental changes.
• Score: 11.737279515161505
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Depth estimation is a core problem in robotic perception and vision tasks, but 3D reconstruction from a single image presents inherent uncertainties. Current depth estimation models primarily rely on inter-image relationships for supervised training, often overlooking the intrinsic information provided by the camera itself. We propose a method that embodies the camera model and its physical characteristics into a deep learning model, computing embodied scene depth through real-time interactions with road environments. The model can calculate embodied scene depth in real-time based on immediate environmental changes using only the intrinsic properties of the camera, without any additional equipment. By combining embodied scene depth with RGB image features, the model gains a comprehensive perspective on both geometric and visual details. Additionally, we incorporate text descriptions containing environmental content and depth information as priors for scene understanding, enriching the model's perception of objects. This integration of image and language - two inherently ambiguous modalities - leverages their complementary strengths for monocular depth estimation. The real-time nature of the embodied language and depth prior model ensures that the model can continuously adjust its perception and behavior in dynamic environments. Experimental results show that the embodied depth estimation method enhances model performance across different scenes.

Related papers

• Leveraging Stable Diffusion for Monocular Depth Estimation via Image Semantic Encoding [1.0445560141983634]
  We propose a novel image-based semantic embedding that extracts contextual information directly from visual features.<n>Our method achieves performance comparable to state-of-the-art models while addressing the shortcomings of CLIP embeddings in handling outdoor scenes.
  arXiv  Detail & Related papers  (2025-02-01T15:37:22Z)
• Structure-Centric Robust Monocular Depth Estimation via Knowledge Distillation [9.032563775151074]
  Monocular depth estimation is a key technique for 3D perception in computer vision. It faces significant challenges in real-world scenarios, which encompass adverse weather variations, motion blur, as well as scenes with poor lighting conditions at night. We devise a novel approach to reduce over-reliance on local textures, enhancing robustness against missing or interfering patterns.
  arXiv  Detail & Related papers  (2024-10-09T15:20:29Z)
• 3D Object Aided Self-Supervised Monocular Depth Estimation [5.579605877061333]
  We propose a new method to address dynamic object movements through monocular 3D object detection. Specifically, we first detect 3D objects in the images and build the per-pixel correspondence of the dynamic pixels with the detected object pose. In this way, the depth of every pixel can be learned via a meaningful geometry model.
  arXiv  Detail & Related papers  (2022-12-04T08:52:33Z)
• SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes [58.89295356901823]
  Self-supervised monocular depth estimation has shown impressive results in static scenes. It relies on the multi-view consistency assumption for training networks, however, that is violated in dynamic object regions. We introduce an external pretrained monocular depth estimation model for generating single-image depth prior. Our model can predict sharp and accurate depth maps, even when training from monocular videos of highly-dynamic scenes.
  arXiv  Detail & Related papers  (2022-11-07T16:17:47Z)
• Monocular Depth Estimation Using Cues Inspired by Biological Vision Systems [22.539300644593936]
  Monocular depth estimation (MDE) aims to transform an RGB image of a scene into a pixelwise depth map from the same camera view. Part of the MDE task is to learn which visual cues in the image can be used for depth estimation, and how. We demonstrate that explicitly injecting visual cue information into the model is beneficial for depth estimation.
  arXiv  Detail & Related papers  (2022-04-21T19:42:36Z)
• Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth [64.29043589521308]
  We propose a rendering module to augment the training data by synthesizing images with virtual-depths. The rendering module takes as input the RGB image and its corresponding sparse depth image, outputs a variety of photo-realistic synthetic images. Besides, we introduce an auxiliary module to improve the detection model by jointly optimizing it through a depth estimation task.
  arXiv  Detail & Related papers  (2021-07-28T11:00:47Z)
• S2R-DepthNet: Learning a Generalizable Depth-specific Structural Representation [63.58891781246175]
  Human can infer the 3D geometry of a scene from a sketch instead of a realistic image, which indicates that the spatial structure plays a fundamental role in understanding the depth of scenes. We are the first to explore the learning of a depth-specific structural representation, which captures the essential feature for depth estimation and ignores irrelevant style information. Our S2R-DepthNet can be well generalized to unseen real-world data directly even though it is only trained on synthetic data.
  arXiv  Detail & Related papers  (2021-04-02T03:55:41Z)
• Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency [114.02182755620784]
  We present an end-to-end joint training framework that explicitly models 6-DoF motion of multiple dynamic objects, ego-motion and depth in a monocular camera setup without supervision. Our framework is shown to outperform the state-of-the-art depth and motion estimation methods.
  arXiv  Detail & Related papers  (2021-02-04T14:26:42Z)
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors [69.02332607843569]
  PriSMONet is a novel approach for learning Multi-Object 3D scene decomposition and representations from single images. A recurrent encoder regresses a latent representation of 3D shape, pose and texture of each object from an input RGB image. We evaluate the accuracy of our model in inferring 3D scene layout, demonstrate its generative capabilities, assess its generalization to real images, and point out benefits of the learned representation.
  arXiv  Detail & Related papers  (2020-10-08T14:49:23Z)
• Novel View Synthesis of Dynamic Scenes with Globally Coherent Depths from a Monocular Camera [93.04135520894631]
  This paper presents a new method to synthesize an image from arbitrary views and times given a collection of images of a dynamic scene. A key challenge for the novel view synthesis arises from dynamic scene reconstruction where epipolar geometry does not apply to the local motion of dynamic contents. To address this challenge, we propose to combine the depth from single view (DSV) and the depth from multi-view stereo (DMV), where DSV is complete, i.e., a depth is assigned to every pixel, yet view-variant in its scale, while DMV is view-invariant yet incomplete.
  arXiv  Detail & Related papers  (2020-04-02T22:45:53Z)
• Shallow2Deep: Indoor Scene Modeling by Single Image Understanding [42.87957414916607]
  We present an automatic indoor scene modeling approach using deep features from neural networks. Given a single RGB image, our method simultaneously recovers semantic contents, 3D geometry and object relationship.
  arXiv  Detail & Related papers  (2020-02-22T23:27:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.