M4Depth: A motion-based approach for monocular depth estimation on video sequences

• URL: http://arxiv.org/abs/2105.09847v2
• Date: Fri, 21 May 2021 09:13:23 GMT
• Title: M4Depth: A motion-based approach for monocular depth estimation on video sequences
• Abstract summary: We present a method to estimate the distance of objects seen by an on-board mounted camera by using its RGB stream and drone motion information. Our network outperforms state-of-the-art depth estimation methods and that the use of motion information is the main contributing factor for this improvement.
• Score: 12.574454799055026
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Getting the distance to objects is crucial for autonomous vehicles. In instances where depth sensors cannot be used, this distance has to be estimated from RGB cameras. As opposed to cars, the task of estimating depth from on-board mounted cameras is made complex on drones because of the lack of constrains on motion during flights. In this paper, we present a method to estimate the distance of objects seen by an on-board mounted camera by using its RGB video stream and drone motion information. Our method is built upon a pyramidal convolutional neural network architecture and uses time recurrence in pair with geometric constraints imposed by motion to produce pixel-wise depth maps. In our architecture, each level of the pyramid is designed to produce its own depth estimate based on past observations and information provided by the previous level in the pyramid. We introduce a spatial reprojection layer to maintain the spatio-temporal consistency of the data between the levels. We analyse the performance of our approach on Mid-Air, a public drone dataset featuring synthetic drone trajectories recorded in a wide variety of unstructured outdoor environments. Our experiments show that our network outperforms state-of-the-art depth estimation methods and that the use of motion information is the main contributing factor for this improvement. The code of our method is publicly available on GitHub; see https://github.com/michael-fonder/M4Depth

Related papers

• Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots [55.43376513158555]
  Camera Depth Models (CDMs) are a simple plugin on daily-use depth cameras.<n>We develop a neural data engine that generates high-quality paired data from simulation by modeling a depth camera's noise pattern.<n>For the first time, our experiments demonstrate, for the first time, that a policy trained on raw simulated depth, without the need for adding noise or real-world fine-tuning, generalizes seamlessly to real-world robots.
  arXiv  Detail & Related papers  (2025-09-02T17:29:38Z)
• Estimating Object Physical Properties from RGB-D Vision and Depth Robot Sensors Using Deep Learning [2.07180164747172]
  Inertial mass plays a crucial role in robotic applications such as object grasping, manipulation, and simulation.<n>This paper proposes a novel approach combining sparse point-cloud data from depth images with RGB images to estimate the mass of objects.
  arXiv  Detail & Related papers  (2025-07-07T14:11:47Z)
• Semi-SD: Semi-Supervised Metric Depth Estimation via Surrounding Cameras for Autonomous Driving [20.19617659712535]
  Semi-SD is a novel metric depth estimation framework tailored for surrounding cameras equipment in autonomous driving. We propose a unified spatial-temporal-semantic fusion module to construct the visual fused features. We evaluate our algorithm on DDAD and nuScenes datasets, and the results demonstrate that our method achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-03-25T14:39:04Z)
• DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos [76.01906393673897]
  We propose a self-supervised method to jointly learn 3D motion and depth from monocular videos. Our system contains a depth estimation module to predict depth, and a new decomposed object-wise 3D motion (DO3D) estimation module to predict ego-motion and 3D object motion. Our model delivers superior performance in all evaluated settings.
  arXiv  Detail & Related papers  (2024-03-09T12:22:46Z)
• Multi-Camera Collaborative Depth Prediction via Consistent Structure Estimation [75.99435808648784]
  We propose a novel multi-camera collaborative depth prediction method. It does not require large overlapping areas while maintaining structure consistency between cameras. Experimental results on DDAD and NuScenes datasets demonstrate the superior performance of our method.
  arXiv  Detail & Related papers  (2022-10-05T03:44:34Z)
• CrossDTR: Cross-view and Depth-guided Transformers for 3D Object Detection [10.696619570924778]
  We propose Cross-view and Depth-guided Transformers for 3D Object Detection, CrossDTR. Our method hugely surpassed existing multi-camera methods by 10 percent in pedestrian detection and about 3 percent in overall mAP and NDS metrics.
  arXiv  Detail & Related papers  (2022-09-27T16:23:12Z)
• ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild [57.37891682117178]
  We present a robust dense indirect structure-from-motion method for videos that is based on dense correspondence from pairwise optical flow. A novel neural network architecture is proposed for processing irregular point trajectory data. Experiments on MPI Sintel dataset show that our system produces significantly more accurate camera trajectories.
  arXiv  Detail & Related papers  (2022-07-19T09:19:45Z)
• Neural Scene Representation for Locomotion on Structured Terrain [56.48607865960868]
  We propose a learning-based method to reconstruct the local terrain for a mobile robot traversing urban environments. Using a stream of depth measurements from the onboard cameras and the robot's trajectory, the estimates the topography in the robot's vicinity. We propose a 3D reconstruction model that faithfully reconstructs the scene, despite the noisy measurements and large amounts of missing data coming from the blind spots of the camera arrangement.
  arXiv  Detail & Related papers  (2022-06-16T10:45:17Z)
• SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation [101.55622133406446]
  We propose a SurroundDepth method to incorporate the information from multiple surrounding views to predict depth maps across cameras. Specifically, we employ a joint network to process all the surrounding views and propose a cross-view transformer to effectively fuse the information from multiple views. In experiments, our method achieves the state-of-the-art performance on the challenging multi-camera depth estimation datasets.
  arXiv  Detail & Related papers  (2022-04-07T17:58:47Z)
• EagerMOT: 3D Multi-Object Tracking via Sensor Fusion [68.8204255655161]
  Multi-object tracking (MOT) enables mobile robots to perform well-informed motion planning and navigation by localizing surrounding objects in 3D space and time. Existing methods rely on depth sensors (e.g., LiDAR) to detect and track targets in 3D space, but only up to a limited sensing range due to the sparsity of the signal. We propose EagerMOT, a simple tracking formulation that integrates all available object observations from both sensor modalities to obtain a well-informed interpretation of the scene dynamics.
  arXiv  Detail & Related papers  (2021-04-29T22:30:29Z)
• Multi-Modal Depth Estimation Using Convolutional Neural Networks [0.8701566919381223]
  This paper addresses the problem of dense depth predictions from sparse distance sensor data and a single camera image on challenging weather conditions. It explores the significance of different sensor modalities such as camera, Radar, and Lidar for estimating depth by applying Deep Learning approaches.
  arXiv  Detail & Related papers  (2020-12-17T15:31:49Z)
• Self-Attention Dense Depth Estimation Network for Unrectified Video Sequences [6.821598757786515]
  LiDAR and radar sensors are the hardware solution for real-time depth estimation. Deep learning based self-supervised depth estimation methods have shown promising results. We propose a self-attention based depth and ego-motion network for unrectified images.
  arXiv  Detail & Related papers  (2020-05-28T21:53:53Z)
• DroTrack: High-speed Drone-based Object Tracking Under Uncertainty [0.23204178451683263]
  DroTrack is a high-speed visual single-object tracking framework for drone-captured video sequences. We implement an effective object segmentation based on Fuzzy C Means. We also leverage the geometrical angular motion to estimate a reliable object scale.
  arXiv  Detail & Related papers  (2020-05-02T13:16:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.