NVDS+: Towards Efficient and Versatile Neural Stabilizer for Video Depth Estimation

• URL: http://arxiv.org/abs/2307.08695v3
• Date: Thu, 03 Oct 2024 17:58:03 GMT
• Title: NVDS+: Towards Efficient and Versatile Neural Stabilizer for Video Depth Estimation
• Authors: Yiran Wang, Min Shi, Jiaqi Li, Chaoyi Hong, Zihao Huang, Juewen Peng, Zhiguo Cao, Jianming Zhang, Ke Xian, Guosheng Lin,
• Abstract summary: Video depth estimation aims to infer temporally consistent depth. We introduce NVDS+ that stabilizes inconsistent depth estimated by various single-image models in a plug-and-play manner. We also elaborate a large-scale Video Depth in the Wild dataset, which contains 14,203 videos with over two million frames.
• Score: 58.21817572577012
• License:
• Abstract: Video depth estimation aims to infer temporally consistent depth. One approach is to finetune a single-image model on each video with geometry constraints, which proves inefficient and lacks robustness. An alternative is learning to enforce consistency from data, which requires well-designed models and sufficient video depth data. To address both challenges, we introduce NVDS+ that stabilizes inconsistent depth estimated by various single-image models in a plug-and-play manner. We also elaborate a large-scale Video Depth in the Wild (VDW) dataset, which contains 14,203 videos with over two million frames, making it the largest natural-scene video depth dataset. Additionally, a bidirectional inference strategy is designed to improve consistency by adaptively fusing forward and backward predictions. We instantiate a model family ranging from small to large scales for different applications. The method is evaluated on VDW dataset and three public benchmarks. To further prove the versatility, we extend NVDS+ to video semantic segmentation and several downstream applications like bokeh rendering, novel view synthesis, and 3D reconstruction. Experimental results show that our method achieves significant improvements in consistency, accuracy, and efficiency. Our work serves as a solid baseline and data foundation for learning-based video depth estimation. Code and dataset are available at: https://github.com/RaymondWang987/NVDS

Related papers

• Video Depth Anything: Consistent Depth Estimation for Super-Long Videos [60.857723250653976]
  We propose Video Depth Anything for high-quality, consistent depth estimation in super-long videos. Our model is trained on a joint dataset of video depth and unlabeled images, similar to Depth Anything V2. Our approach sets a new state-of-the-art in zero-shot video depth estimation.
  arXiv  Detail & Related papers  (2025-01-21T18:53:30Z)
• Align3R: Aligned Monocular Depth Estimation for Dynamic Videos [50.28715151619659]
  We propose a novel video-depth estimation method called Align3R to estimate temporal consistent depth maps for a dynamic video. Our key idea is to utilize the recent DUSt3R model to align estimated monocular depth maps of different timesteps. Experiments demonstrate that Align3R estimates consistent video depth and camera poses for a monocular video with superior performance than baseline methods.
  arXiv  Detail & Related papers  (2024-12-04T07:09:59Z)
• Video Depth without Video Models [34.11454612504574]
  Video depth estimation lifts monocular video clips to 3D by inferring dense depth at every frame. We show how to turn a single-image latent diffusion model (LDM) into a state-of-the-art video depth estimator. Our model, which we call RollingDepth, has two main ingredients: (i) a multi-frame depth estimator that is derived from a single-image LDM and maps very short video snippets to depth snippets.
  arXiv  Detail & Related papers  (2024-11-28T14:50:14Z)
• Depth Any Video with Scalable Synthetic Data [98.42356740981839]
  We develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse synthetic environments. We leverage the powerful priors of generative video diffusion models to handle real-world videos effectively. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.
  arXiv  Detail & Related papers  (2024-10-14T17:59:46Z)
• SM4Depth: Seamless Monocular Metric Depth Estimation across Multiple Cameras and Scenes by One Model [72.0795843450604]
  Current approaches face challenges in maintaining consistent accuracy across diverse scenes. These methods rely on extensive datasets comprising millions, if not tens of millions, of data for training. This paper presents SM$4$Depth, a model that seamlessly works for both indoor and outdoor scenes.
  arXiv  Detail & Related papers  (2024-03-13T14:08:25Z)
• FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models [67.96827539201071]
  We propose a novel test-time optimization approach for 3D scene reconstruction. Our method achieves state-of-the-art cross-dataset reconstruction on five zero-shot testing datasets.
  arXiv  Detail & Related papers  (2023-08-10T17:55:02Z)
• ViDaS Video Depth-aware Saliency Network [40.08270905030302]
  We introduce ViDaS, a two-stream, fully convolutional Video, Depth-Aware Saliency network. It addresses the problem of attention modeling in-the-wild" via saliency prediction in videos. Network consists of two visual streams, one for the RGB frames, and one for the depth frames. It is trained end-to-end and is evaluated in a variety of different databases with eye-tracking data.
  arXiv  Detail & Related papers  (2023-05-19T15:04:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.